2024-04-09T21:22:38.941 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-09T21:22:38.942 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917 --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} -- /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/orig.config.yaml 2024-04-09T21:22:38.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-09T21:22:39.065 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917 branch: reef-release description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7647917' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef-release ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef-release sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 6358 sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6JmiQup3jBbUc4fDfxA1Cc87n92MyW1E37W5w047Bu63ZvbHk+9o3MyQjRiYMiRUTXFQavy0sy2i0bbdnktEc= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/q3UvERIx3e325Ay3goK8EiCuOC295xGHManCHbnvw67Gvt8s2Nmjbhmw1zIntD9LFijFSyuVpPD5AtVlfbmU= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-09_01:21:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-09T21:22:39.066 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa; will attempt to use it 2024-04-09T21:22:39.066 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks 2024-04-09T21:22:39.067 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-09T21:22:39.068 INFO:teuthology.task.internal:Checking packages... 2024-04-09T21:22:39.093 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'd540ebaca6b131a1dd560e7f69e024b133bbaa42' 2024-04-09T21:22:39.093 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-09T21:22:39.093 INFO:teuthology.packaging:ref: None 2024-04-09T21:22:39.093 INFO:teuthology.packaging:tag: None 2024-04-09T21:22:39.093 INFO:teuthology.packaging:branch: reef-release 2024-04-09T21:22:39.093 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-09T21:22:39.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2024-04-09T21:22:39.368 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1187.gd540ebac 2024-04-09T21:22:39.370 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-09T21:22:39.376 INFO:teuthology.task.internal:no buildpackages task found 2024-04-09T21:22:39.376 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-09T21:22:39.419 INFO:teuthology.task.internal:Saving configuration 2024-04-09T21:22:39.432 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-09T21:22:39.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-09T21:22:39.469 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-09 21:17:27.270827', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6JmiQup3jBbUc4fDfxA1Cc87n92MyW1E37W5w047Bu63ZvbHk+9o3MyQjRiYMiRUTXFQavy0sy2i0bbdnktEc='} 2024-04-09T21:22:39.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-09 21:17:27.272267', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/q3UvERIx3e325Ay3goK8EiCuOC295xGHManCHbnvw67Gvt8s2Nmjbhmw1zIntD9LFijFSyuVpPD5AtVlfbmU='} 2024-04-09T21:22:39.489 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-09T21:22:39.496 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-04-09T21:22:39.496 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-04-09T21:22:39.496 INFO:teuthology.run_tasks:Running task console_log... 2024-04-09T21:22:39.579 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6603966e50>, signals=[15]) 2024-04-09T21:22:39.580 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-09T21:22:39.587 INFO:teuthology.task.internal:Opening connections... 2024-04-09T21:22:39.587 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-04-09T21:22:39.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-09T21:22:39.664 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-04-09T21:22:39.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-09T21:22:39.738 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-09T21:22:39.745 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-04-09T21:22:39.766 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-04-09T21:22:39.767 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-04-09T21:22:39.827 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="Red Hat Enterprise Linux" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="8.6 (Ootpa)" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:ID="rhel" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="fedora" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="8.6" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el8" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://www.redhat.com/" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:39.828 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-09T21:22:39.829 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-09T21:22:39.829 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-09T21:22:39.829 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-09T21:22:39.829 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-04-09T21:22:39.849 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-04-09T21:22:39.870 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-04-09T21:22:39.870 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Red Hat Enterprise Linux" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8.6 (Ootpa)" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:ID="rhel" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="fedora" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8.6" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-09T21:22:39.929 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.redhat.com/" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-09T21:22:39.930 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-09T21:22:39.930 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-04-09T21:22:39.949 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-09T21:22:39.959 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-09T21:22:39.986 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-09T21:22:39.986 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-04-09T21:22:39.988 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-04-09T21:22:40.007 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-09T21:22:40.013 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-09T21:22:40.013 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-04-09T21:22:40.048 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-04-09T21:22:40.080 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-09T21:22:40.132 INFO:teuthology.run_tasks:Running task kernel... 2024-04-09T21:22:40.146 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-09T21:22:40.146 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-09T21:22:40.146 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-09T21:22:40.147 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-09T21:22:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-09T21:22:40.165 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-04-09T21:22:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-09T21:22:40.167 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-04-09T21:22:40.223 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:22:40.223 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:22:40.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-04-09T21:22:40.227 INFO:teuthology.orchestra.run.smithi063.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:22:40.227 INFO:teuthology.task.kernel:Running kernel on smithi063: 4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:22:40.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-04-09T21:22:41.674 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:22:41.675 INFO:teuthology.orchestra.run.smithi063.stdout:Unable to read consumer identity 2024-04-09T21:22:41.696 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:22:41.697 INFO:teuthology.orchestra.run.smithi137.stdout:Unable to read consumer identity 2024-04-09T21:22:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:22:41.962 INFO:teuthology.orchestra.run.smithi137.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-09T21:22:41.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:22:41.968 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:41.969 INFO:teuthology.orchestra.run.smithi063.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-09T21:22:41.969 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:43.613 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-04-09T21:22:43.861 INFO:teuthology.orchestra.run.smithi063.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-04-09T21:22:51.193 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 16 MB 00:06 2024-04-09T21:22:55.032 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 481 kB/s | 24 kB 00:00 2024-04-09T21:22:55.809 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-09T21:22:55.809 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-09T21:22:55.890 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:22:55.891 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:22:55.891 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:22:56.020 DEBUG:teuthology.orchestra.run.smithi063:> echo no | sudo yum reinstall kernel || true 2024-04-09T21:22:56.378 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:22:56.378 INFO:teuthology.orchestra.run.smithi063.stdout:Unable to read consumer identity 2024-04-09T21:22:56.595 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:56.596 INFO:teuthology.orchestra.run.smithi063.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-09T21:22:56.596 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:56.921 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:02 ago on Tue 09 Apr 2024 09:22:54 PM UTC. 2024-04-09T21:22:56.994 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-09T21:22:57.010 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-09T21:22:57.047 INFO:teuthology.orchestra.run.smithi063.stderr:Error: No packages marked for reinstall. 2024-04-09T21:22:57.095 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum reinstall -y kernel || true 2024-04-09T21:22:57.450 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:22:57.450 INFO:teuthology.orchestra.run.smithi063.stdout:Unable to read consumer identity 2024-04-09T21:22:57.757 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:57.757 INFO:teuthology.orchestra.run.smithi063.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-09T21:22:57.757 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:22:58.073 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:04 ago on Tue 09 Apr 2024 09:22:54 PM UTC. 2024-04-09T21:22:58.144 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-09T21:22:58.145 INFO:teuthology.orchestra.run.smithi063.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-09T21:22:58.176 INFO:teuthology.orchestra.run.smithi063.stderr:Error: No packages marked for reinstall. 2024-04-09T21:22:58.224 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-04-09T21:22:58.397 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:22:58.397 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:22:58.397 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-09T21:22:58.397 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-04-09T21:22:58.398 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-09T21:22:58.428 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-04-09T21:22:58.442 DEBUG:teuthology.parallel:result is None 2024-04-09T21:23:03.271 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 849 kB/s | 16 MB 00:19 2024-04-09T21:23:06.877 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 483 kB/s | 24 kB 00:00 2024-04-09T21:23:07.638 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-09T21:23:07.639 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-09T21:23:07.722 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:23:07.723 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:23:07.723 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:23:07.867 DEBUG:teuthology.orchestra.run.smithi137:> echo no | sudo yum reinstall kernel || true 2024-04-09T21:23:08.203 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:23:08.203 INFO:teuthology.orchestra.run.smithi137.stdout:Unable to read consumer identity 2024-04-09T21:23:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:23:08.450 INFO:teuthology.orchestra.run.smithi137.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-09T21:23:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:23:08.759 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:02 ago on Tue 09 Apr 2024 09:23:06 PM UTC. 2024-04-09T21:23:08.841 INFO:teuthology.orchestra.run.smithi137.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-09T21:23:08.841 INFO:teuthology.orchestra.run.smithi137.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-09T21:23:08.883 INFO:teuthology.orchestra.run.smithi137.stderr:Error: No packages marked for reinstall. 2024-04-09T21:23:08.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum reinstall -y kernel || true 2024-04-09T21:23:09.264 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:23:09.264 INFO:teuthology.orchestra.run.smithi137.stdout:Unable to read consumer identity 2024-04-09T21:23:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:23:09.420 INFO:teuthology.orchestra.run.smithi137.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-09T21:23:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:23:09.728 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:03 ago on Tue 09 Apr 2024 09:23:06 PM UTC. 2024-04-09T21:23:09.794 INFO:teuthology.orchestra.run.smithi137.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-09T21:23:09.794 INFO:teuthology.orchestra.run.smithi137.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-09T21:23:09.830 INFO:teuthology.orchestra.run.smithi137.stderr:Error: No packages marked for reinstall. 2024-04-09T21:23:09.883 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-04-09T21:23:10.036 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:23:10.037 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-09T21:23:10.037 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-09T21:23:10.037 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-04-09T21:23:10.037 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-09T21:23:10.080 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-04-09T21:23:10.095 DEBUG:teuthology.parallel:result is None 2024-04-09T21:23:10.096 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-09T21:23:10.103 INFO:teuthology.task.internal:Creating test directory... 2024-04-09T21:23:10.104 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-09T21:23:10.106 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-09T21:23:10.126 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-09T21:23:10.133 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-09T21:23:10.166 INFO:teuthology.task.internal:Creating archive directory... 2024-04-09T21:23:10.166 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-09T21:23:10.169 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-09T21:23:10.200 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-09T21:23:10.207 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-09T21:23:10.208 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-09T21:23:10.238 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-09T21:23:10.278 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-09T21:23:10.280 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-09T21:23:10.288 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-09T21:23:10.292 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-09T21:23:10.293 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-09T21:23:10.302 INFO:teuthology.task.internal:Configuring sudo... 2024-04-09T21:23:10.302 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-09T21:23:10.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-09T21:23:10.357 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-09T21:23:10.367 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-09T21:23:10.367 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-09T21:23:10.374 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-09T21:23:10.414 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-09T21:23:10.477 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-09T21:23:10.535 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-09T21:23:10.577 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-09T21:23:10.623 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:23:10.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-09T21:23:10.692 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-09T21:23:10.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-09T21:23:10.782 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-09T21:23:10.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-09T21:23:10.907 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:23:10.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-09T21:23:10.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-04-09T21:23:10.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-04-09T21:23:11.025 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-09T21:23:11.057 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-09T21:23:11.522 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-09T21:23:11.554 INFO:teuthology.task.internal:Starting timer... 2024-04-09T21:23:11.554 INFO:teuthology.run_tasks:Running task pcp... 2024-04-09T21:23:11.576 INFO:teuthology.run_tasks:Running task selinux... 2024-04-09T21:23:11.617 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-09T21:23:11.617 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-04-09T21:23:11.690 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-04-09T21:23:11.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-04-09T21:23:11.740 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-04-09T21:23:11.742 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-09T21:23:11.742 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-04-09T21:23:11.767 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-04-09T21:23:11.768 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-04-09T21:23:11.789 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-04-09T21:23:11.789 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-04-09T21:23:11.789 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-09T21:23:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-09T21:23:11.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-09T21:23:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-09T21:23:11.863 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-09T21:23:11.863 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-04-09T21:23:11.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-04-09T21:23:11.960 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-09T21:23:11.974 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-09T21:23:11.990 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-09T21:23:11.991 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-09T21:32:52.885 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-04-09T21:32:52.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-09T21:32:52.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-09T21:32:52.966 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-04-09T21:32:53.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-09T21:32:53.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-09T21:32:53.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-09T21:32:53.116 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-04-09T21:32:53.143 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-04-09T21:32:53.144 INFO:teuthology.run_tasks:Running task clock... 2024-04-09T21:32:53.154 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-09T21:32:53.155 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-09T21:32:53.155 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-09T21:32:53.158 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-09T21:32:53.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-09T21:32:53.190 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-09T21:32:53.205 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-09T21:32:53.214 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-09T21:32:53.229 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-09T21:32:53.290 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-04-09T21:32:53.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-04-09T21:32:53.302 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-04-09T21:32:53.311 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-04-09T21:32:53.317 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-09T21:32:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-09T21:32:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-09T21:32:53.340 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-09T21:32:53.402 INFO:teuthology.orchestra.run.smithi063.stderr:bash: ntpq: command not found 2024-04-09T21:32:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-09T21:32:53.405 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-04-09T21:32:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.406 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.418 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2024-04-09T21:32:53.421 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-09T21:32:53.421 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-04-09T21:32:53.422 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.422 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.422 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.422 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-09T21:32:53.423 INFO:teuthology.run_tasks:Running task install... 2024-04-09T21:32:53.431 DEBUG:teuthology.task.install:project ceph 2024-04-09T21:32:53.431 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}} 2024-04-09T21:32:53.431 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-09T21:32:53.431 INFO:teuthology.task.install:Using flavor: default 2024-04-09T21:32:53.437 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-09T21:32:53.437 INFO:teuthology.task.install:extra packages: [] 2024-04-09T21:32:53.438 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-09T21:32:53.438 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-09T21:32:53.440 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-09T21:32:53.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-09T21:32:53.661 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/ 2024-04-09T21:32:53.662 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-09T21:32:53.666 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/ 2024-04-09T21:32:53.666 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-09T21:32:53.834 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-09T21:32:53.834 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:32:53.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-09T21:32:53.840 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-09T21:32:53.841 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:32:53.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-09T21:32:53.863 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-09T21:32:53.863 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-09T21:32:53.868 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-09T21:32:53.868 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-09T21:32:53.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-09T21:32:53.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-09T21:32:54.012 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-09T21:32:54.014 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-09T21:32:54.079 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-04-09T21:32:54.080 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-04-09T21:32:54.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-04-09T21:32:54.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-04-09T21:32:54.458 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:32:54.476 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:32:55.877 INFO:teuthology.orchestra.run.smithi063.stdout:61 files removed 2024-04-09T21:32:55.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-04-09T21:32:55.987 INFO:teuthology.orchestra.run.smithi137.stdout:61 files removed 2024-04-09T21:32:56.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-04-09T21:32:56.289 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:32:56.372 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:32:58.305 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 245 kB/s | 80 kB 00:00 2024-04-09T21:32:58.388 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 218 kB/s | 80 kB 00:00 2024-04-09T21:32:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-04-09T21:32:58.747 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-04-09T21:32:58.989 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2024-04-09T21:32:59.148 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 4.0 kB/s | 1.5 kB 00:00 2024-04-09T21:32:59.148 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-09T21:32:59.307 INFO:teuthology.orchestra.run.smithi063.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-04-09T21:33:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-04-09T21:33:00.601 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-04-09T21:33:03.624 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 353 kB/s | 24 kB 00:00 2024-04-09T21:33:04.259 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 16 MB/s | 8.8 MB 00:00 2024-04-09T21:33:04.374 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 414 kB/s | 24 kB 00:00 2024-04-09T21:33:05.801 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 67 MB 00:01 2024-04-09T21:33:07.627 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 MB/s | 67 MB 00:01 2024-04-09T21:33:17.009 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 12 MB/s | 8.8 MB 00:00 2024-04-09T21:33:18.428 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 51 MB/s | 60 MB 00:01 2024-04-09T21:33:20.202 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 60 MB 00:01 2024-04-09T21:33:34.030 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout:==================================================================================================== 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout:==================================================================================================== 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 13 M 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.7 M 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.6 M 2024-04-09T21:33:34.033 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 5.5 M 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 22 M 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 24 k 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 780 k 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 474 k 2024-04-09T21:33:34.034 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.6 M 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 45 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 129 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 197 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 372 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 364 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 107 k 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout:==================================================================================================== 2024-04-09T21:33:34.035 INFO:teuthology.orchestra.run.smithi137.stdout:Install 21 Packages 2024-04-09T21:33:34.036 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-04-09T21:33:34.036 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:33:34.036 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 59 M 2024-04-09T21:33:34.036 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:33:35.377 INFO:teuthology.orchestra.run.smithi137.stdout:(1/23): ceph-radosgw-18.2.2-1187.gd540ebac.el8. 10 MB/s | 13 MB 00:01 2024-04-09T21:33:35.444 INFO:teuthology.orchestra.run.smithi137.stdout:(2/23): ceph-selinux-18.2.2-1187.gd540ebac.el8. 365 kB/s | 24 kB 00:00 2024-04-09T21:33:35.535 INFO:teuthology.orchestra.run.smithi137.stdout:(3/23): libcephfs2-18.2.2-1187.gd540ebac.el8.x8 8.3 MB/s | 780 kB 00:00 2024-04-09T21:33:35.619 INFO:teuthology.orchestra.run.smithi137.stdout:(4/23): libradosstriper1-18.2.2-1187.gd540ebac. 5.6 MB/s | 474 kB 00:00 2024-04-09T21:33:35.828 INFO:teuthology.orchestra.run.smithi137.stdout:(5/23): ceph-common-18.2.2-1187.gd540ebac.el8.x 12 MB/s | 22 MB 00:01 2024-04-09T21:33:35.895 INFO:teuthology.orchestra.run.smithi137.stdout:(6/23): python3-ceph-argparse-18.2.2-1187.gd540 671 kB/s | 45 kB 00:00 2024-04-09T21:33:35.953 INFO:teuthology.orchestra.run.smithi137.stdout:(7/23): python3-ceph-common-18.2.2-1187.gd540eb 2.2 MB/s | 129 kB 00:00 2024-04-09T21:33:36.029 INFO:teuthology.orchestra.run.smithi137.stdout:(8/23): python3-cephfs-18.2.2-1187.gd540ebac.el 2.6 MB/s | 197 kB 00:00 2024-04-09T21:33:36.104 INFO:teuthology.orchestra.run.smithi137.stdout:(9/23): librgw2-18.2.2-1187.gd540ebac.el8.x86_6 14 MB/s | 6.6 MB 00:00 2024-04-09T21:33:36.129 INFO:teuthology.orchestra.run.smithi137.stdout:(10/23): python3-rados-18.2.2-1187.gd540ebac.el 3.6 MB/s | 372 kB 00:00 2024-04-09T21:33:36.154 INFO:teuthology.orchestra.run.smithi137.stdout:(11/23): python3-rbd-18.2.2-1187.gd540ebac.el8. 7.1 MB/s | 364 kB 00:00 2024-04-09T21:33:36.180 INFO:teuthology.orchestra.run.smithi137.stdout:(12/23): python3-rgw-18.2.2-1187.gd540ebac.el8. 2.1 MB/s | 107 kB 00:00 2024-04-09T21:33:36.263 INFO:teuthology.orchestra.run.smithi137.stdout:(13/23): ceph-base-18.2.2-1187.gd540ebac.el8.x8 2.5 MB/s | 5.5 MB 00:02 2024-04-09T21:33:36.655 INFO:teuthology.orchestra.run.smithi137.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 192 kB/s | 75 kB 00:00 2024-04-09T21:33:36.739 INFO:teuthology.orchestra.run.smithi137.stdout:(15/23): liboath-2.6.2-3.el8.x86_64.rpm 105 kB/s | 59 kB 00:00 2024-04-09T21:33:36.926 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:33:36.928 INFO:teuthology.orchestra.run.smithi063.stdout:==================================================================================================== 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout:==================================================================================================== 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 13 M 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.7 M 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.6 M 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 5.5 M 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 22 M 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 24 k 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 780 k 2024-04-09T21:33:36.929 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 474 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.6 M 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 45 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 129 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 197 k 2024-04-09T21:33:36.930 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 372 k 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 364 k 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 107 k 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout:==================================================================================================== 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout:Install 21 Packages 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-04-09T21:33:36.931 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:33:36.932 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 59 M 2024-04-09T21:33:36.932 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:33:36.939 INFO:teuthology.orchestra.run.smithi137.stdout:(16/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 235 kB/s | 47 kB 00:00 2024-04-09T21:33:36.989 INFO:teuthology.orchestra.run.smithi137.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 883 kB/s | 44 kB 00:00 2024-04-09T21:33:37.148 INFO:teuthology.orchestra.run.smithi137.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2024-04-09T21:33:37.198 INFO:teuthology.orchestra.run.smithi137.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 293 kB/s | 306 kB 00:01 2024-04-09T21:33:37.315 INFO:teuthology.orchestra.run.smithi137.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.1 MB/s | 354 kB 00:00 2024-04-09T21:33:38.427 INFO:teuthology.orchestra.run.smithi063.stdout:(1/23): ceph-radosgw-18.2.2-1187.gd540ebac.el8. 9.2 MB/s | 13 MB 00:01 2024-04-09T21:33:38.466 INFO:teuthology.orchestra.run.smithi137.stdout:(21/23): librbd1-18.2.2-1187.gd540ebac.el8.x86_ 3.1 MB/s | 3.6 MB 00:01 2024-04-09T21:33:38.499 INFO:teuthology.orchestra.run.smithi063.stdout:(2/23): ceph-selinux-18.2.2-1187.gd540ebac.el8. 344 kB/s | 24 kB 00:00 2024-04-09T21:33:38.595 INFO:teuthology.orchestra.run.smithi063.stdout:(3/23): libcephfs2-18.2.2-1187.gd540ebac.el8.x8 8.0 MB/s | 780 kB 00:00 2024-04-09T21:33:38.678 INFO:teuthology.orchestra.run.smithi063.stdout:(4/23): libradosstriper1-18.2.2-1187.gd540ebac. 5.6 MB/s | 474 kB 00:00 2024-04-09T21:33:38.799 INFO:teuthology.orchestra.run.smithi137.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 828 kB/s | 1.7 MB 00:02 2024-04-09T21:33:38.970 INFO:teuthology.orchestra.run.smithi063.stdout:(5/23): ceph-common-18.2.2-1187.gd540ebac.el8.x 11 MB/s | 22 MB 00:01 2024-04-09T21:33:39.029 INFO:teuthology.orchestra.run.smithi063.stdout:(6/23): python3-ceph-argparse-18.2.2-1187.gd540 767 kB/s | 45 kB 00:00 2024-04-09T21:33:39.087 INFO:teuthology.orchestra.run.smithi063.stdout:(7/23): python3-ceph-common-18.2.2-1187.gd540eb 2.2 MB/s | 129 kB 00:00 2024-04-09T21:33:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:(8/23): librgw2-18.2.2-1187.gd540ebac.el8.x86_6 13 MB/s | 6.6 MB 00:00 2024-04-09T21:33:39.196 INFO:teuthology.orchestra.run.smithi063.stdout:(9/23): python3-cephfs-18.2.2-1187.gd540ebac.el 1.8 MB/s | 197 kB 00:00 2024-04-09T21:33:39.199 INFO:teuthology.orchestra.run.smithi137.stdout:(23/23): librados2-18.2.2-1187.gd540ebac.el8.x8 1.8 MB/s | 3.7 MB 00:02 2024-04-09T21:33:39.201 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:33:39.202 INFO:teuthology.orchestra.run.smithi137.stdout:Total 11 MB/s | 59 MB 00:05 2024-04-09T21:33:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:(10/23): python3-rados-18.2.2-1187.gd540ebac.el 7.3 MB/s | 372 kB 00:00 2024-04-09T21:33:39.246 INFO:teuthology.orchestra.run.smithi063.stdout:(11/23): python3-rbd-18.2.2-1187.gd540ebac.el8. 7.1 MB/s | 364 kB 00:00 2024-04-09T21:33:39.252 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:33:39.263 INFO:teuthology.orchestra.run.smithi063.stdout:(12/23): python3-rgw-18.2.2-1187.gd540ebac.el8. 3.1 MB/s | 107 kB 00:00 2024-04-09T21:33:39.298 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:33:39.298 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:33:39.372 INFO:teuthology.orchestra.run.smithi063.stdout:(13/23): ceph-base-18.2.2-1187.gd540ebac.el8.x8 2.3 MB/s | 5.5 MB 00:02 2024-04-09T21:33:39.629 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:33:39.638 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:33:39.639 INFO:teuthology.orchestra.run.smithi063.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 157 kB/s | 59 kB 00:00 2024-04-09T21:33:39.714 INFO:teuthology.orchestra.run.smithi063.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 219 kB/s | 75 kB 00:00 2024-04-09T21:33:39.798 INFO:teuthology.orchestra.run.smithi063.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 555 kB/s | 306 kB 00:00 2024-04-09T21:33:39.848 INFO:teuthology.orchestra.run.smithi063.stdout:(17/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 353 kB/s | 47 kB 00:00 2024-04-09T21:33:39.948 INFO:teuthology.orchestra.run.smithi063.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2024-04-09T21:33:39.974 INFO:teuthology.orchestra.run.smithi063.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 251 kB/s | 44 kB 00:00 2024-04-09T21:33:40.082 INFO:teuthology.orchestra.run.smithi063.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.6 MB/s | 354 kB 00:00 2024-04-09T21:33:40.124 INFO:teuthology.orchestra.run.smithi063.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.6 MB/s | 1.7 MB 00:00 2024-04-09T21:33:40.366 INFO:teuthology.orchestra.run.smithi063.stdout:(22/23): librbd1-18.2.2-1187.gd540ebac.el8.x86_ 13 MB/s | 3.6 MB 00:00 2024-04-09T21:33:40.783 INFO:teuthology.orchestra.run.smithi063.stdout:(23/23): librados2-18.2.2-1187.gd540ebac.el8.x8 4.6 MB/s | 3.7 MB 00:00 2024-04-09T21:33:40.785 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:33:40.785 INFO:teuthology.orchestra.run.smithi063.stdout:Total 15 MB/s | 59 MB 00:03 2024-04-09T21:33:40.830 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:33:40.834 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:33:40.884 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:33:40.884 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:33:41.005 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-09T21:33:41.155 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-09T21:33:41.235 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:33:41.244 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:33:41.558 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-09T21:33:41.788 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-09T21:33:41.889 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-09T21:33:42.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-09T21:33:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-09T21:33:42.472 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:33:42.529 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-09T21:33:42.657 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-09T21:33:42.805 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-09T21:33:42.919 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-09T21:33:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-09T21:33:43.232 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-09T21:33:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-09T21:33:43.446 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-09T21:33:43.490 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-09T21:33:43.514 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-09T21:33:43.592 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-09T21:33:43.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-09T21:33:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-09T21:33:43.865 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 10/25 2024-04-09T21:33:43.990 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-09T21:33:44.065 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-09T21:33:44.188 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-09T21:33:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-09T21:33:44.429 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-09T21:33:44.487 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-09T21:33:44.623 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-09T21:33:44.646 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-09T21:33:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-09T21:33:44.846 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-09T21:33:44.908 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-09T21:33:45.071 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-09T21:33:45.122 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-09T21:33:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-09T21:33:45.635 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-09T21:33:45.659 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 10/25 2024-04-09T21:33:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-09T21:33:45.851 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-09T21:33:45.884 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-09T21:33:46.084 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-09T21:33:46.114 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 18/25 2024-04-09T21:33:46.254 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 19/25 2024-04-09T21:33:46.271 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-09T21:33:46.347 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-09T21:33:46.498 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-09T21:33:46.683 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-09T21:33:47.530 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-09T21:33:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-09T21:33:47.793 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-09T21:33:48.043 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 18/25 2024-04-09T21:33:48.229 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 19/25 2024-04-09T21:33:49.909 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-09T21:33:50.145 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-09T21:33:50.736 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-09T21:33:50.994 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-09T21:33:51.078 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-09T21:33:51.258 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-09T21:33:52.167 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-09T21:33:52.387 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-09T21:33:53.018 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-09T21:33:53.286 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-09T21:33:53.374 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-09T21:33:53.558 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-09T21:34:10.640 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /run 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2024-04-09T21:34:10.641 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:11.966 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-09T21:34:11.987 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-09T21:34:11.988 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-09T21:34:12.194 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-09T21:34:12.194 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-09T21:34:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 1/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 5/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 6/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 8/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 9/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 10/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-09T21:34:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 13/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-09T21:34:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 24/25 2024-04-09T21:34:13.506 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /dev 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /run 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /dev 2024-04-09T21:34:13.507 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:14.917 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-09T21:34:14.939 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-09T21:34:14.939 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-09T21:34:15.187 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-09T21:34:15.187 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-09T21:34:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 1/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 5/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 6/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 8/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 9/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 10/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-09T21:34:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 13/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 24/25 2024-04-09T21:34:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-09T21:34:20.035 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-09T21:34:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-09T21:34:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:20.590 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:34:20.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-04-09T21:34:21.125 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:34:23.024 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 21 kB/s | 4.1 kB 00:00 2024-04-09T21:34:23.375 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-09T21:34:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-09T21:34:25.706 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:34:25.745 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:34:25.746 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-04-09T21:34:25.746 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 47 M 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 142 k 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-09T21:34:25.747 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 48 M 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 213 M 2024-04-09T21:34:25.748 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:34:25.777 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.4 MB/s | 70 kB 00:00 2024-04-09T21:34:25.953 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2024-04-09T21:34:26.053 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): libcephsqlite-18.2.2-1187.gd540ebac.el8. 467 kB/s | 142 kB 00:00 2024-04-09T21:34:26.128 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2024-04-09T21:34:26.221 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-04-09T21:34:26.296 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:26.296 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-04-09T21:34:26.296 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.296 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.296 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-09T21:34:26.297 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:26.298 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:34:26.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-04-09T21:34:26.873 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:34:28.037 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el8.x86_ 21 MB/s | 47 MB 00:02 2024-04-09T21:34:28.039 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:34:28.039 INFO:teuthology.orchestra.run.smithi137.stdout:Total 21 MB/s | 48 MB 00:02 2024-04-09T21:34:28.079 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:34:28.100 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:34:28.100 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:34:28.370 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:34:28.379 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:34:28.770 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-09T21:34:28.871 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:34:29.005 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-09T21:34:29.082 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-09T21:34:29.226 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-09T21:34:29.283 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-09T21:34:29.489 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-09T21:34:29.677 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-09T21:34:29.813 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-09T21:34:32.012 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 47 M 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:34:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 142 k 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-04-09T21:34:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 48 M 2024-04-09T21:34:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 213 M 2024-04-09T21:34:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:34:32.052 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2024-04-09T21:34:32.208 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): libcephsqlite-18.2.2-1187.gd540ebac.el8. 736 kB/s | 142 kB 00:00 2024-04-09T21:34:32.242 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2024-04-09T21:34:32.417 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 971 kB/s | 202 kB 00:00 2024-04-09T21:34:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 981 kB/s | 188 kB 00:00 2024-04-09T21:34:34.363 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-09T21:34:34.459 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el8.x86_ 19 MB/s | 47 MB 00:02 2024-04-09T21:34:34.461 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:34:34.461 INFO:teuthology.orchestra.run.smithi063.stdout:Total 20 MB/s | 48 MB 00:02 2024-04-09T21:34:34.501 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:34:34.524 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:34:34.524 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:34:34.609 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-09T21:34:34.809 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:34:34.818 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:34:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-09T21:34:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 1/6 2024-04-09T21:34:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2/6 2024-04-09T21:34:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-09T21:34:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-09T21:34:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-09T21:34:35.298 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:34:35.457 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-09T21:34:35.521 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-09T21:34:35.729 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-09T21:34:35.925 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-09T21:34:36.108 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-09T21:34:36.281 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-09T21:34:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-09T21:34:41.348 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-09T21:34:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-09T21:34:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 1/6 2024-04-09T21:34:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2/6 2024-04-09T21:34:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-09T21:34:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-09T21:34:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-09T21:34:44.775 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-09T21:34:44.775 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-7.el8.x86_64 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:45.311 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:34:45.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-04-09T21:34:45.808 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:34:47.659 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 kB/s | 4.1 kB 00:00 2024-04-09T21:34:48.032 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-09T21:34:50.432 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:34:50.435 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================= 2024-04-09T21:34:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:34:50.435 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================= 2024-04-09T21:34:50.435 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:34:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.4 k 2024-04-09T21:34:50.435 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 2.3 M 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 1.6 M 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 245 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 4.8 M 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 18 M 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-09T21:34:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-09T21:34:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================= 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout:Install 32 Packages 2024-04-09T21:34:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:34:50.439 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 M 2024-04-09T21:34:50.439 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 124 M 2024-04-09T21:34:50.439 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:34:50.896 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-18.2.2-1187.gd540ebac.el8.x86_64.r 56 kB/s | 6.4 kB 00:00 2024-04-09T21:34:51.031 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-09T21:34:51.031 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:34:51.215 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-mgr-18.2.2-1187.gd540ebac.el8.x86_ 3.6 MB/s | 1.6 MB 00:00 2024-04-09T21:34:51.297 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): ceph-mon-18.2.2-1187.gd540ebac.el8.x86_ 12 MB/s | 4.8 MB 00:00 2024-04-09T21:34:51.347 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1187.gd540 4.8 MB/s | 245 kB 00:00 2024-04-09T21:34:51.397 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): ceph-mds-18.2.2-1187.gd540ebac.el8.x86_ 3.7 MB/s | 2.3 MB 00:00 2024-04-09T21:34:51.610 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: jq-1.6-7.el8.x86_64 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:51.611 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:34:51.622 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 158 kB/s | 44 kB 00:00 2024-04-09T21:34:51.781 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 488 kB/s | 185 kB 00:00 2024-04-09T21:34:51.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-04-09T21:34:51.806 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 939 kB/s | 173 kB 00:00 2024-04-09T21:34:51.882 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-04-09T21:34:51.957 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.1 MB/s | 384 kB 00:00 2024-04-09T21:34:51.982 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-04-09T21:34:52.032 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 647 kB/s | 49 kB 00:00 2024-04-09T21:34:52.066 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 701 kB/s | 59 kB 00:00 2024-04-09T21:34:52.125 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-09T21:34:52.141 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:34:52.150 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 187 kB/s | 16 kB 00:00 2024-04-09T21:34:52.201 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-04-09T21:34:52.226 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-04-09T21:34:52.276 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 389 kB/s | 29 kB 00:00 2024-04-09T21:34:52.302 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-04-09T21:34:52.393 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-04-09T21:34:52.418 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.8 MB/s | 255 kB 00:00 2024-04-09T21:34:52.477 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-04-09T21:34:52.503 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 274 kB/s | 23 kB 00:00 2024-04-09T21:34:52.653 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 623 kB/s | 109 kB 00:00 2024-04-09T21:34:52.703 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2024-04-09T21:34:52.795 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): python3-cryptography-3.2.1-7.el8_9.x86 3.9 MB/s | 559 kB 00:00 2024-04-09T21:34:52.837 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.3 MB/s | 457 kB 00:00 2024-04-09T21:34:52.887 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 423 kB/s | 39 kB 00:00 2024-04-09T21:34:52.971 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 774 kB/s | 103 kB 00:00 2024-04-09T21:34:53.038 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2024-04-09T21:34:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): ceph-osd-18.2.2-1187.gd540ebac.el8.x86 9.1 MB/s | 18 MB 00:02 2024-04-09T21:34:53.271 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 181 kB/s | 54 kB 00:00 2024-04-09T21:34:53.275 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:34:53.276 INFO:teuthology.orchestra.run.smithi137.stdout:Total 11 MB/s | 31 MB 00:02 2024-04-09T21:34:53.375 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:34:53.413 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:34:53.413 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:34:53.638 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:34:53.647 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:34:53.961 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-09T21:34:54.423 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-09T21:34:54.751 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:34:54.947 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-09T21:34:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-09T21:34:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-09T21:34:55.469 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-09T21:34:55.750 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-09T21:34:55.954 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-09T21:34:56.144 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-09T21:34:56.532 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-09T21:34:56.786 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-09T21:34:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-09T21:34:57.126 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-09T21:34:57.216 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================= 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================= 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.4 k 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:34:57.223 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 2.3 M 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 1.6 M 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 245 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 4.8 M 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 18 M 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-09T21:34:57.224 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-09T21:34:57.225 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================= 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:Install 32 Packages 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 31 M 2024-04-09T21:34:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 124 M 2024-04-09T21:34:57.227 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:34:57.363 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-09T21:34:57.468 INFO:teuthology.orchestra.run.smithi063.stdout:(1/32): ceph-18.2.2-1187.gd540ebac.el8.x86_64.r 35 kB/s | 6.4 kB 00:00 2024-04-09T21:34:57.587 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-09T21:34:57.618 INFO:teuthology.orchestra.run.smithi063.stdout:(2/32): ceph-mgr-18.2.2-1187.gd540ebac.el8.x86_ 4.7 MB/s | 1.6 MB 00:00 2024-04-09T21:34:57.668 INFO:teuthology.orchestra.run.smithi063.stdout:(3/32): ceph-mds-18.2.2-1187.gd540ebac.el8.x86_ 5.9 MB/s | 2.3 MB 00:00 2024-04-09T21:34:57.743 INFO:teuthology.orchestra.run.smithi063.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1187.gd540 3.2 MB/s | 245 kB 00:00 2024-04-09T21:34:57.824 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-09T21:34:57.980 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-09T21:34:58.044 INFO:teuthology.orchestra.run.smithi063.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 145 kB/s | 44 kB 00:00 2024-04-09T21:34:58.171 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-09T21:34:58.270 INFO:teuthology.orchestra.run.smithi063.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 823 kB/s | 185 kB 00:00 2024-04-09T21:34:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-09T21:34:58.511 INFO:teuthology.orchestra.run.smithi063.stdout:(7/32): ceph-osd-18.2.2-1187.gd540ebac.el8.x86_ 21 MB/s | 18 MB 00:00 2024-04-09T21:34:58.520 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-09T21:34:58.536 INFO:teuthology.orchestra.run.smithi063.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 646 kB/s | 173 kB 00:00 2024-04-09T21:34:58.636 INFO:teuthology.orchestra.run.smithi063.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-04-09T21:34:58.705 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-09T21:34:58.728 INFO:teuthology.orchestra.run.smithi063.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2024-04-09T21:34:58.820 INFO:teuthology.orchestra.run.smithi063.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 532 kB/s | 49 kB 00:00 2024-04-09T21:34:58.896 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-09T21:34:58.921 INFO:teuthology.orchestra.run.smithi063.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 586 kB/s | 59 kB 00:00 2024-04-09T21:34:58.971 INFO:teuthology.orchestra.run.smithi063.stdout:(13/32): python3-cherrypy-18.4.0-1.el8.noarch.r 837 kB/s | 384 kB 00:00 2024-04-09T21:34:59.013 INFO:teuthology.orchestra.run.smithi063.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-09T21:34:59.067 INFO:teuthology.orchestra.run.smithi063.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2024-04-09T21:34:59.073 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-09T21:34:59.105 INFO:teuthology.orchestra.run.smithi063.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-04-09T21:34:59.155 INFO:teuthology.orchestra.run.smithi063.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 260 kB/s | 24 kB 00:00 2024-04-09T21:34:59.222 INFO:teuthology.orchestra.run.smithi063.stdout:(18/32): ceph-mon-18.2.2-1187.gd540ebac.el8.x86 2.8 MB/s | 4.8 MB 00:01 2024-04-09T21:34:59.273 INFO:teuthology.orchestra.run.smithi063.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 175 kB/s | 29 kB 00:00 2024-04-09T21:34:59.298 INFO:teuthology.orchestra.run.smithi063.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 193 kB/s | 27 kB 00:00 2024-04-09T21:34:59.314 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-09T21:34:59.365 INFO:teuthology.orchestra.run.smithi063.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-04-09T21:34:59.390 INFO:teuthology.orchestra.run.smithi063.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 930 kB/s | 85 kB 00:00 2024-04-09T21:34:59.465 INFO:teuthology.orchestra.run.smithi063.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 228 kB/s | 23 kB 00:00 2024-04-09T21:34:59.524 INFO:teuthology.orchestra.run.smithi063.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 817 kB/s | 109 kB 00:00 2024-04-09T21:34:59.556 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-09T21:34:59.666 INFO:teuthology.orchestra.run.smithi063.stdout:(25/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2024-04-09T21:34:59.691 INFO:teuthology.orchestra.run.smithi063.stdout:(26/32): python3-waitress-1.4.3-1.el8.noarch.rp 545 kB/s | 255 kB 00:00 2024-04-09T21:34:59.725 INFO:teuthology.orchestra.run.smithi063.stdout:(27/32): python3-cryptography-3.2.1-7.el8_9.x86 2.7 MB/s | 559 kB 00:00 2024-04-09T21:34:59.769 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-09T21:34:59.967 INFO:teuthology.orchestra.run.smithi063.stdout:(28/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-04-09T21:34:59.992 INFO:teuthology.orchestra.run.smithi063.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 129 kB/s | 39 kB 00:00 2024-04-09T21:35:00.034 INFO:teuthology.orchestra.run.smithi063.stdout:(30/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 345 kB/s | 103 kB 00:00 2024-04-09T21:35:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-09T21:35:00.167 INFO:teuthology.orchestra.run.smithi063.stdout:(31/32): python3-pytz-2017.2-11.el8.noarch.rpm 310 kB/s | 54 kB 00:00 2024-04-09T21:35:00.193 INFO:teuthology.orchestra.run.smithi063.stdout:(32/32): python3-mako-1.0.6-14.el8.noarch.rpm 696 kB/s | 157 kB 00:00 2024-04-09T21:35:00.194 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:35:00.194 INFO:teuthology.orchestra.run.smithi063.stdout:Total 10 MB/s | 31 MB 00:02 2024-04-09T21:35:00.296 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:35:00.336 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:35:00.337 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:35:00.382 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-09T21:35:00.578 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:35:00.587 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:35:00.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 27/32 2024-04-09T21:35:00.932 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-09T21:35:01.710 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:35:01.930 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-09T21:35:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-09T21:35:02.300 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-09T21:35:02.527 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-09T21:35:02.579 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-09T21:35:02.843 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-09T21:35:02.957 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-09T21:35:03.051 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-09T21:35:03.284 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-09T21:35:03.436 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-09T21:35:03.617 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-09T21:35:03.675 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-09T21:35:03.831 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-09T21:35:03.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-09T21:35:04.033 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-09T21:35:04.052 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-09T21:35:04.145 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-09T21:35:04.173 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-09T21:35:04.313 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-09T21:35:04.559 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-09T21:35:04.580 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-09T21:35:04.580 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 1/32 2024-04-09T21:35:04.580 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2/32 2024-04-09T21:35:04.580 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 3/32 2024-04-09T21:35:04.580 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 4/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 5/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 6/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-09T21:35:04.581 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-09T21:35:04.582 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-09T21:35:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-09T21:35:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-09T21:35:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-09T21:35:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-09T21:35:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-09T21:35:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-09T21:35:04.793 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-09T21:35:04.993 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-09T21:35:05.174 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-09T21:35:05.356 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-09T21:35:05.525 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-09T21:35:05.685 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-09T21:35:05.903 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-09T21:35:06.132 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-09T21:35:06.326 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-09T21:35:06.580 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-09T21:35:06.827 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-09T21:35:07.081 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-09T21:35:07.369 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-09T21:35:07.691 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-09T21:35:08.074 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 27/32 2024-04-09T21:35:08.268 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-09T21:35:10.020 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-09T21:35:11.443 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-09T21:35:11.957 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-09T21:35:12.119 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-09T21:35:12.346 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-09T21:35:12.521 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-09T21:35:12.541 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-09T21:35:12.651 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-09T21:35:12.651 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:35:12.660 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 1/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 3/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 4/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 5/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 6/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-09T21:35:13.073 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-09T21:35:13.074 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-09T21:35:13.075 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-09T21:35:13.075 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-09T21:35:13.075 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-09T21:35:13.075 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-09T21:35:13.075 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-09T21:35:13.075 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-09T21:35:13.076 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-09T21:35:13.076 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-09T21:35:13.198 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-2.6-1.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-09T21:35:13.199 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:13.200 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:35:13.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-04-09T21:35:13.739 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:15.559 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-09T21:35:15.882 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-04-09T21:35:18.163 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:35:18.350 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:35:18.351 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:35:18.351 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:35:18.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-04-09T21:35:18.869 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:20.749 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 21 kB/s | 4.1 kB 00:00 2024-04-09T21:35:21.118 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-09T21:35:21.185 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-09T21:35:21.185 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:35:21.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:21.780 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:35:21.780 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:21.780 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:21.780 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:21.780 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-09T21:35:21.781 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-2.6-1.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-09T21:35:21.782 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:21.783 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:35:21.968 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-04-09T21:35:22.340 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:23.597 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:35:23.597 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:35:23.597 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:35:23.597 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:35:23.597 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 170 k 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:23.598 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 170 k 2024-04-09T21:35:23.599 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 692 k 2024-04-09T21:35:23.599 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:35:23.892 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-18.2.2-1187.gd540ebac.el8.noarch.rpm 580 kB/s | 170 kB 00:00 2024-04-09T21:35:23.893 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:35:23.893 INFO:teuthology.orchestra.run.smithi137.stdout:Total 578 kB/s | 170 kB 00:00 2024-04-09T21:35:23.893 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:35:23.901 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:35:23.901 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:35:23.903 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:35:23.912 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:35:23.966 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:35:24.284 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-09T21:35:24.653 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 52 kB/s | 4.5 kB 00:00 2024-04-09T21:35:25.687 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:25.841 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:26.045 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:27.232 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:35:27.430 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:35:27.431 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:35:27.431 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:35:27.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-04-09T21:35:27.945 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:29.751 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-09T21:35:30.194 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-09T21:35:32.999 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:35:33.000 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:35:33.000 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 170 k 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 170 k 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 692 k 2024-04-09T21:35:33.001 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:35:33.299 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-18.2.2-1187.gd540ebac.el8.noarch.rpm 571 kB/s | 170 kB 00:00 2024-04-09T21:35:33.300 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:35:33.300 INFO:teuthology.orchestra.run.smithi063.stdout:Total 569 kB/s | 170 kB 00:00 2024-04-09T21:35:33.300 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:35:33.309 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:35:33.309 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:35:33.311 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:35:33.321 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:35:33.369 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:35:33.999 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:34.000 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:35:35.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:35.133 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:35:35.133 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:35:35.133 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:35.133 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:35:35.204 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:35.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-04-09T21:35:35.378 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:35.586 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:35.660 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:37.458 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-09T21:35:37.854 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-09T21:35:40.336 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 165 k 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:35:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:40.338 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 165 k 2024-04-09T21:35:40.338 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 503 k 2024-04-09T21:35:40.338 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:35:40.625 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 577 kB/s | 165 kB 00:00 2024-04-09T21:35:40.625 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:35:40.626 INFO:teuthology.orchestra.run.smithi137.stdout:Total 574 kB/s | 165 kB 00:00 2024-04-09T21:35:40.626 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:35:40.637 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:35:40.637 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:35:40.691 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:35:40.700 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:35:40.818 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:35:40.994 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-09T21:35:41.393 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-09T21:35:44.406 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-09T21:35:44.407 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:35:44.993 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:44.993 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:35:44.993 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:35:44.993 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:44.994 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:35:45.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-04-09T21:35:45.555 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:47.406 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:35:47.857 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-09T21:35:50.640 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 165 k 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:35:50.641 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:35:50.642 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:50.642 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 165 k 2024-04-09T21:35:50.642 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 503 k 2024-04-09T21:35:50.642 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:35:50.825 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 904 kB/s | 165 kB 00:00 2024-04-09T21:35:50.826 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:35:50.826 INFO:teuthology.orchestra.run.smithi063.stdout:Total 898 kB/s | 165 kB 00:00 2024-04-09T21:35:50.826 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:35:50.838 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:35:50.838 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:35:50.896 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:35:50.905 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:35:51.025 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:35:51.164 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-09T21:35:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-09T21:35:51.425 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:35:51.552 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-09T21:35:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:52.008 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:35:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:35:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:35:52.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-04-09T21:35:52.518 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:54.261 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:35:54.593 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-09T21:35:56.778 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:35:56.968 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:35:56.969 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:35:56.969 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:35:57.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-04-09T21:35:57.447 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:35:59.324 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-09T21:35:59.367 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-09T21:35:59.367 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:35:59.695 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-09T21:35:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:59.951 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:35:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:35:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:35:59.951 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:36:00.110 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-04-09T21:36:00.482 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:36:02.073 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:36:02.074 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:36:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 3.5 M 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 24 k 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 15 k 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout:Install 5 Packages 2024-04-09T21:36:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:02.076 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.8 M 2024-04-09T21:36:02.076 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 87 M 2024-04-09T21:36:02.076 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:36:02.361 INFO:teuthology.orchestra.run.smithi137.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1187.gd540 95 kB/s | 15 kB 00:00 2024-04-09T21:36:02.411 INFO:teuthology.orchestra.run.smithi137.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1187.gd54 118 kB/s | 24 kB 00:00 2024-04-09T21:36:02.432 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-09T21:36:02.629 INFO:teuthology.orchestra.run.smithi137.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 8.4 MB/s | 3.5 MB 00:00 2024-04-09T21:36:02.653 INFO:teuthology.orchestra.run.smithi137.stdout:(4/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 115 kB/s | 34 kB 00:00 2024-04-09T21:36:02.745 INFO:teuthology.orchestra.run.smithi137.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 589 kB/s | 196 kB 00:00 2024-04-09T21:36:02.747 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:36:02.747 INFO:teuthology.orchestra.run.smithi137.stdout:Total 5.6 MB/s | 3.8 MB 00:00 2024-04-09T21:36:02.748 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:36:02.755 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:36:02.755 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:36:02.824 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:36:02.833 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:36:02.898 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 24 kB/s | 4.5 kB 00:00 2024-04-09T21:36:03.054 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:36:03.279 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-09T21:36:03.483 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-09T21:36:03.596 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-09T21:36:04.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 4/5 2024-04-09T21:36:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-09T21:36:05.209 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-09T21:36:05.209 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 1/5 2024-04-09T21:36:05.209 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 2/5 2024-04-09T21:36:05.209 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-09T21:36:05.210 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-09T21:36:05.474 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:36:05.676 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:36:05.678 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:36:05.678 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:36:05.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-04-09T21:36:06.194 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:36:07.999 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-09T21:36:08.352 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 kB/s | 4.5 kB 00:00 2024-04-09T21:36:11.138 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 3.5 M 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 24 k 2024-04-09T21:36:11.139 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 15 k 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout:Install 5 Packages 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:11.140 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.8 M 2024-04-09T21:36:11.141 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 87 M 2024-04-09T21:36:11.141 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:36:11.480 INFO:teuthology.orchestra.run.smithi063.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1187.gd54 165 kB/s | 24 kB 00:00 2024-04-09T21:36:11.539 INFO:teuthology.orchestra.run.smithi063.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1187.gd540 71 kB/s | 15 kB 00:00 2024-04-09T21:36:11.631 INFO:teuthology.orchestra.run.smithi063.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 224 kB/s | 34 kB 00:00 2024-04-09T21:36:11.697 INFO:teuthology.orchestra.run.smithi063.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 9.7 MB/s | 3.5 MB 00:00 2024-04-09T21:36:11.765 INFO:teuthology.orchestra.run.smithi063.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 872 kB/s | 196 kB 00:00 2024-04-09T21:36:11.766 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:36:11.766 INFO:teuthology.orchestra.run.smithi063.stdout:Total 6.1 MB/s | 3.8 MB 00:00 2024-04-09T21:36:11.766 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:36:11.775 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:36:11.775 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:36:11.849 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:36:11.858 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:36:12.090 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:36:12.311 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-09T21:36:12.536 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-09T21:36:12.668 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-09T21:36:13.900 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 4/5 2024-04-09T21:36:14.101 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-09T21:36:14.382 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-09T21:36:14.382 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 1/5 2024-04-09T21:36:14.383 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 2/5 2024-04-09T21:36:14.383 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-09T21:36:14.383 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-09T21:36:16.635 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-09T21:36:16.635 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:36:17.174 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:17.174 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:36:17.174 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:17.174 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:17.174 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:17.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-09T21:36:17.175 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-09T21:36:17.175 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:17.175 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:36:17.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-09T21:36:17.715 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:36:19.767 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-09T21:36:20.105 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-09T21:36:22.473 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================================= 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================================= 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 7.4 M 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-09T21:36:22.475 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================================= 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout:Install 8 Packages 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:22.476 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 36 M 2024-04-09T21:36:22.477 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 207 M 2024-04-09T21:36:22.477 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:36:22.660 INFO:teuthology.orchestra.run.smithi137.stdout:(1/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2024-04-09T21:36:22.685 INFO:teuthology.orchestra.run.smithi137.stdout:(2/8): libquadmath-8.5.0-20.el8.x86_64.rpm 830 kB/s | 172 kB 00:00 2024-04-09T21:36:22.935 INFO:teuthology.orchestra.run.smithi137.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 901 kB/s | 225 kB 00:00 2024-04-09T21:36:23.036 INFO:teuthology.orchestra.run.smithi137.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 9.9 MB/s | 3.7 MB 00:00 2024-04-09T21:36:23.161 INFO:teuthology.orchestra.run.smithi137.stdout:(5/8): ceph-mgr-diskprediction-local-18.2.2-118 11 MB/s | 7.4 MB 00:00 2024-04-09T21:36:23.279 INFO:teuthology.orchestra.run.smithi137.stdout:(6/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 15 MB/s | 5.2 MB 00:00 2024-04-09T21:36:23.478 INFO:teuthology.orchestra.run.smithi137.stdout:(7/8): openblas-0.3.15-6.el8.x86_64.rpm 11 MB/s | 5.0 MB 00:00 2024-04-09T21:36:23.695 INFO:teuthology.orchestra.run.smithi137.stdout:(8/8): python3-scipy-1.0.0-21.module+el8.9.0+20 26 MB/s | 14 MB 00:00 2024-04-09T21:36:23.696 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:36:23.696 INFO:teuthology.orchestra.run.smithi137.stdout:Total 30 MB/s | 36 MB 00:01 2024-04-09T21:36:23.945 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:36:23.963 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:36:23.963 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:36:24.274 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:36:24.282 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:36:24.821 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-09T21:36:24.821 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:36:24.830 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:36:24.986 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-09T21:36:25.116 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-09T21:36:25.269 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-09T21:36:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:25.426 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:36:25.426 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-09T21:36:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-09T21:36:25.427 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:25.427 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:36:25.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-09T21:36:25.903 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-09T21:36:25.968 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:36:26.053 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-09T21:36:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-09T21:36:26.861 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-09T21:36:27.549 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-09T21:36:27.843 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:36:27.890 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-09T21:36:28.086 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-09T21:36:28.277 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-09T21:36:29.844 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-09T21:36:31.010 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:36:31.011 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================================= 2024-04-09T21:36:31.011 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:36:31.011 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================================= 2024-04-09T21:36:31.011 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:36:31.011 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 7.4 M 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================================= 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout:Install 8 Packages 2024-04-09T21:36:31.012 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:31.013 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 36 M 2024-04-09T21:36:31.013 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 207 M 2024-04-09T21:36:31.013 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:36:31.121 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-09T21:36:31.163 INFO:teuthology.orchestra.run.smithi063.stdout:(1/8): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-04-09T21:36:31.188 INFO:teuthology.orchestra.run.smithi063.stdout:(2/8): libquadmath-8.5.0-20.el8.x86_64.rpm 989 kB/s | 172 kB 00:00 2024-04-09T21:36:31.310 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-09T21:36:31.331 INFO:teuthology.orchestra.run.smithi063.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-04-09T21:36:31.422 INFO:teuthology.orchestra.run.smithi063.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2024-04-09T21:36:31.564 INFO:teuthology.orchestra.run.smithi063.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.2 MB 00:00 2024-04-09T21:36:31.610 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-09T21:36:31.610 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 1/8 2024-04-09T21:36:31.610 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-09T21:36:31.610 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-09T21:36:31.611 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-09T21:36:31.611 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-09T21:36:31.611 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-09T21:36:31.611 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-09T21:36:31.673 INFO:teuthology.orchestra.run.smithi063.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-04-09T21:36:31.923 INFO:teuthology.orchestra.run.smithi063.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 38 MB/s | 14 MB 00:00 2024-04-09T21:36:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-118 7.4 MB/s | 7.4 MB 00:01 2024-04-09T21:36:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:36:32.016 INFO:teuthology.orchestra.run.smithi063.stdout:Total 36 MB/s | 36 MB 00:01 2024-04-09T21:36:32.269 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:36:32.288 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:36:32.288 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:36:32.616 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:36:32.626 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:36:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:36:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-09T21:36:33.470 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-09T21:36:33.669 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-09T21:36:34.322 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-09T21:36:34.509 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-09T21:36:35.144 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-09T21:36:35.327 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-09T21:36:36.056 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-09T21:36:36.384 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-09T21:36:36.619 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-09T21:36:38.499 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-09T21:36:38.866 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-09T21:36:38.867 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:39.420 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:36:39.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-04-09T21:36:39.779 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-09T21:36:39.954 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:36:39.957 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 1/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-09T21:36:40.260 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-09T21:36:41.758 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-09T21:36:42.051 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 kB/s | 4.5 kB 00:00 2024-04-09T21:36:44.546 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================== 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================== 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 49 k 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-09T21:36:44.548 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:36:44.549 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================== 2024-04-09T21:36:44.550 INFO:teuthology.orchestra.run.smithi137.stdout:Install 14 Packages 2024-04-09T21:36:44.550 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:44.550 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-04-09T21:36:44.550 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 25 M 2024-04-09T21:36:44.550 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:36:44.869 INFO:teuthology.orchestra.run.smithi137.stdout:(1/14): ceph-mgr-rook-18.2.2-1187.gd540ebac.el8 196 kB/s | 49 kB 00:00 2024-04-09T21:36:44.953 INFO:teuthology.orchestra.run.smithi137.stdout:(2/14): python3-certifi-2018.10.15-7.el8.noarch 49 kB/s | 16 kB 00:00 2024-04-09T21:36:44.978 INFO:teuthology.orchestra.run.smithi137.stdout:(3/14): python3-cachetools-3.1.1-4.el8.noarch.r 92 kB/s | 33 kB 00:00 2024-04-09T21:36:45.170 INFO:teuthology.orchestra.run.smithi137.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 335 kB/s | 64 kB 00:00 2024-04-09T21:36:45.295 INFO:teuthology.orchestra.run.smithi137.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 490 kB/s | 61 kB 00:00 2024-04-09T21:36:45.320 INFO:teuthology.orchestra.run.smithi137.stdout:(6/14): python3-google-auth-1.1.1-10.el8.noarch 180 kB/s | 81 kB 00:00 2024-04-09T21:36:45.488 INFO:teuthology.orchestra.run.smithi137.stdout:(7/14): python3-kubernetes-11.0.0-6.el8.noarch. 2.2 MB/s | 1.2 MB 00:00 2024-04-09T21:36:45.521 INFO:teuthology.orchestra.run.smithi137.stdout:(8/14): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2024-04-09T21:36:45.546 INFO:teuthology.orchestra.run.smithi137.stdout:(9/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 687 kB/s | 155 kB 00:00 2024-04-09T21:36:45.647 INFO:teuthology.orchestra.run.smithi137.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 269 kB/s | 43 kB 00:00 2024-04-09T21:36:45.688 INFO:teuthology.orchestra.run.smithi137.stdout:(11/14): python3-pyasn1-modules-0.3.7-6.el8.noa 776 kB/s | 110 kB 00:00 2024-04-09T21:36:45.722 INFO:teuthology.orchestra.run.smithi137.stdout:(12/14): python3-jsonpointer-1.10-11.el8.noarch 102 kB/s | 20 kB 00:00 2024-04-09T21:36:45.756 INFO:teuthology.orchestra.run.smithi137.stdout:(13/14): python3-jsonpatch-1.21-2.el8.noarch.rp 244 kB/s | 27 kB 00:00 2024-04-09T21:36:45.781 INFO:teuthology.orchestra.run.smithi137.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2024-04-09T21:36:45.783 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:36:45.784 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-04-09T21:36:45.861 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:36:45.877 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:36:45.877 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:36:45.959 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:36:45.967 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:36:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-09T21:36:46.943 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-09T21:36:47.139 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-09T21:36:47.281 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-09T21:36:47.450 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-09T21:36:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-09T21:36:47.970 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-09T21:36:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-09T21:36:48.439 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-09T21:36:48.637 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-09T21:36:48.885 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-09T21:36:49.926 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-09T21:36:50.184 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-09T21:36:50.340 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 1/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-09T21:36:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-09T21:36:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-09T21:36:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-09T21:36:50.820 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-09T21:36:50.820 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:36:51.415 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:51.416 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:36:51.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-04-09T21:36:51.957 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:36:53.841 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-09T21:36:54.274 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-09T21:36:57.052 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:36:57.054 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================== 2024-04-09T21:36:57.054 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:36:57.054 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================== 2024-04-09T21:36:57.054 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:36:57.054 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 49 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-09T21:36:57.055 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================== 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout:Install 14 Packages 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.0 M 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 25 M 2024-04-09T21:36:57.056 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:36:57.174 INFO:teuthology.orchestra.run.smithi063.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 223 kB/s | 16 kB 00:00 2024-04-09T21:36:57.199 INFO:teuthology.orchestra.run.smithi063.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 335 kB/s | 33 kB 00:00 2024-04-09T21:36:57.224 INFO:teuthology.orchestra.run.smithi063.stdout:(3/14): python3-google-auth-1.1.1-10.el8.noarch 1.7 MB/s | 81 kB 00:00 2024-04-09T21:36:57.258 INFO:teuthology.orchestra.run.smithi063.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 1.9 MB/s | 64 kB 00:00 2024-04-09T21:36:57.292 INFO:teuthology.orchestra.run.smithi063.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 1.8 MB/s | 61 kB 00:00 2024-04-09T21:36:57.325 INFO:teuthology.orchestra.run.smithi063.stdout:(6/14): python3-kubernetes-11.0.0-6.el8.noarch. 9.3 MB/s | 1.2 MB 00:00 2024-04-09T21:36:57.392 INFO:teuthology.orchestra.run.smithi063.stdout:(7/14): ceph-mgr-rook-18.2.2-1187.gd540ebac.el8 169 kB/s | 49 kB 00:00 2024-04-09T21:36:57.493 INFO:teuthology.orchestra.run.smithi063.stdout:(8/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 975 kB/s | 155 kB 00:00 2024-04-09T21:36:57.534 INFO:teuthology.orchestra.run.smithi063.stdout:(9/14): python3-jwt-1.6.1-2.el8.noarch.rpm 176 kB/s | 43 kB 00:00 2024-04-09T21:36:57.576 INFO:teuthology.orchestra.run.smithi063.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 232 kB/s | 43 kB 00:00 2024-04-09T21:36:57.627 INFO:teuthology.orchestra.run.smithi063.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 149 kB/s | 20 kB 00:00 2024-04-09T21:36:57.677 INFO:teuthology.orchestra.run.smithi063.stdout:(12/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2024-04-09T21:36:57.702 INFO:teuthology.orchestra.run.smithi063.stdout:(13/14): python3-pyasn1-modules-0.3.7-6.el8.noa 656 kB/s | 110 kB 00:00 2024-04-09T21:36:57.719 INFO:teuthology.orchestra.run.smithi063.stdout:(14/14): python3-jsonpatch-1.21-2.el8.noarch.rp 186 kB/s | 27 kB 00:00 2024-04-09T21:36:57.720 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:36:57.720 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-04-09T21:36:57.794 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:36:57.811 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:36:57.811 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:36:57.898 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:36:57.907 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:36:57.916 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-09T21:36:57.916 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:36:58.422 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-09T21:36:58.482 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:36:58.483 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:36:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-09T21:36:58.902 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-09T21:36:59.089 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-09T21:36:59.263 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-09T21:36:59.449 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-09T21:36:59.709 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-09T21:36:59.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-04-09T21:36:59.987 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-09T21:37:00.210 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-09T21:37:00.283 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:00.430 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-09T21:37:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-09T21:37:00.858 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-09T21:37:01.986 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-09T21:37:02.124 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-09T21:37:02.217 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-09T21:37:02.388 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-09T21:37:02.504 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-09T21:37:02.668 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-09T21:37:02.668 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 1/14 2024-04-09T21:37:02.668 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-09T21:37:02.668 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-09T21:37:02.668 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-09T21:37:02.668 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-09T21:37:02.669 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-09T21:37:04.942 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================================== 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================================== 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 138 k 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-09T21:37:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:04.944 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:37:04.944 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================================== 2024-04-09T21:37:04.944 INFO:teuthology.orchestra.run.smithi137.stdout:Install 5 Packages 2024-04-09T21:37:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:04.944 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 6.0 M 2024-04-09T21:37:04.944 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-04-09T21:37:04.945 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:37:05.369 INFO:teuthology.orchestra.run.smithi137.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 163 kB/s | 60 kB 00:00 2024-04-09T21:37:05.394 INFO:teuthology.orchestra.run.smithi137.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 349 kB/s | 138 kB 00:00 2024-04-09T21:37:05.511 INFO:teuthology.orchestra.run.smithi137.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 929 kB/s | 476 kB 00:00 2024-04-09T21:37:05.686 INFO:teuthology.orchestra.run.smithi137.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2024-04-09T21:37:05.795 INFO:teuthology.orchestra.run.smithi137.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-04-09T21:37:05.797 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:05.797 INFO:teuthology.orchestra.run.smithi137.stdout:Total 7.0 MB/s | 6.0 MB 00:00 2024-04-09T21:37:05.855 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:37:05.867 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:37:05.867 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:37:05.956 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:37:05.965 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:37:06.917 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:37:07.250 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-09T21:37:07.498 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-09T21:37:07.794 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-09T21:37:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-09T21:37:08.279 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-09T21:37:08.561 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-09T21:37:08.561 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/5 2024-04-09T21:37:08.562 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-09T21:37:08.562 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-09T21:37:08.562 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-09T21:37:10.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-09T21:37:10.365 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:37:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:10.979 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:37:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:37:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-09T21:37:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-09T21:37:10.979 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:37:11.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-04-09T21:37:11.544 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:13.531 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:37:13.956 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-09T21:37:16.756 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:37:16.757 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================================== 2024-04-09T21:37:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:37:16.757 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================================== 2024-04-09T21:37:16.757 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 138 k 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================================== 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout:Install 5 Packages 2024-04-09T21:37:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:16.759 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 6.0 M 2024-04-09T21:37:16.759 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-04-09T21:37:16.759 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:37:17.117 INFO:teuthology.orchestra.run.smithi063.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 184 kB/s | 60 kB 00:00 2024-04-09T21:37:17.142 INFO:teuthology.orchestra.run.smithi063.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 392 kB/s | 138 kB 00:00 2024-04-09T21:37:17.268 INFO:teuthology.orchestra.run.smithi063.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 996 kB/s | 476 kB 00:00 2024-04-09T21:37:17.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-09T21:37:17.314 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:37:17.376 INFO:teuthology.orchestra.run.smithi063.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.2 MB/s | 538 kB 00:00 2024-04-09T21:37:17.460 INFO:teuthology.orchestra.run.smithi063.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2024-04-09T21:37:17.461 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:17.461 INFO:teuthology.orchestra.run.smithi063.stdout:Total 8.5 MB/s | 6.0 MB 00:00 2024-04-09T21:37:17.516 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:37:17.529 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:37:17.529 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:37:17.624 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:37:17.633 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:37:17.888 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:17.889 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:37:18.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-04-09T21:37:18.415 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:18.677 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:37:18.991 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-09T21:37:19.231 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-09T21:37:19.561 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-09T21:37:19.855 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-09T21:37:20.013 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-09T21:37:20.279 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-09T21:37:20.280 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/5 2024-04-09T21:37:20.280 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-09T21:37:20.280 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-09T21:37:20.280 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-09T21:37:20.366 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-09T21:37:20.708 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 859 k 2024-04-09T21:37:23.080 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 942 k 2024-04-09T21:37:23.081 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 3.0 M 2024-04-09T21:37:23.082 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:37:23.338 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 324 kB/s | 83 kB 00:00 2024-04-09T21:37:23.363 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el8.x86_ 3.0 MB/s | 859 kB 00:00 2024-04-09T21:37:23.364 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:23.364 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.2 MB/s | 942 kB 00:00 2024-04-09T21:37:23.376 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:37:23.398 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:37:23.398 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:37:23.482 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:37:23.491 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:37:23.657 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:37:23.918 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-09T21:37:24.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-09T21:37:24.496 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-09T21:37:24.496 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/2 2024-04-09T21:37:30.116 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-09T21:37:30.116 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:37:30.721 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:30.721 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:37:30.721 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:37:30.722 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-09T21:37:30.722 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-09T21:37:30.722 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-09T21:37:30.722 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-09T21:37:30.722 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:30.722 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:37:30.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-04-09T21:37:31.307 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:33.251 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2024-04-09T21:37:33.697 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-09T21:37:34.520 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-09T21:37:34.520 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:37:35.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:35.100 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:37:35.100 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-09T21:37:35.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:35.100 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:37:35.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-04-09T21:37:35.633 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:36.518 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:37:36.518 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:37:36.518 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:37:36.518 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout: x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 859 k 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 942 k 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 3.0 M 2024-04-09T21:37:36.519 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:37:36.671 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 544 kB/s | 83 kB 00:00 2024-04-09T21:37:36.913 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el8.x86_ 2.1 MB/s | 859 kB 00:00 2024-04-09T21:37:36.914 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:36.914 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.3 MB/s | 942 kB 00:00 2024-04-09T21:37:36.926 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:37:36.949 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:37:36.949 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:37:37.040 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:37:37.049 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:37:37.221 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:37:37.456 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:37:37.471 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-09T21:37:37.634 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-09T21:37:37.758 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-09T21:37:38.023 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-09T21:37:38.024 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/2 2024-04-09T21:37:40.148 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================== 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================== 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 263 k 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-09T21:37:40.149 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:40.150 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:37:40.150 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================================== 2024-04-09T21:37:40.150 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-04-09T21:37:40.150 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:40.150 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 311 k 2024-04-09T21:37:40.150 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.4 M 2024-04-09T21:37:40.151 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:37:40.351 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 241 kB/s | 48 kB 00:00 2024-04-09T21:37:40.467 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el8.no 830 kB/s | 263 kB 00:00 2024-04-09T21:37:40.468 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:40.468 INFO:teuthology.orchestra.run.smithi137.stdout:Total 978 kB/s | 311 kB 00:00 2024-04-09T21:37:40.480 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:37:40.492 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:37:40.492 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:37:40.530 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:37:40.538 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:37:40.666 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:37:40.963 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-09T21:37:41.248 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-09T21:37:41.648 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-09T21:37:41.648 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 1/2 2024-04-09T21:37:49.621 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-09T21:37:49.621 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:37:50.222 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:50.222 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:37:50.223 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-09T21:37:50.223 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:50.223 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:37:50.393 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-volume 2024-04-09T21:37:50.775 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:50.941 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-09T21:37:50.942 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:37:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:51.495 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:37:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:37:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-09T21:37:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:51.495 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:37:51.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-04-09T21:37:52.022 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:37:52.805 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-09T21:37:53.221 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-09T21:37:53.966 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-09T21:37:54.286 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-09T21:37:56.007 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:37:56.007 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================== 2024-04-09T21:37:56.007 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-09T21:37:56.007 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================== 2024-04-09T21:37:56.007 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:37:56.007 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 263 k 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================================== 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:37:56.008 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 311 k 2024-04-09T21:37:56.009 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.4 M 2024-04-09T21:37:56.009 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:37:56.195 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 260 kB/s | 48 kB 00:00 2024-04-09T21:37:56.220 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el8.no 1.2 MB/s | 263 kB 00:00 2024-04-09T21:37:56.221 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:56.221 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.4 MB/s | 311 kB 00:00 2024-04-09T21:37:56.233 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:37:56.245 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:37:56.245 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:37:56.286 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:37:56.295 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:37:56.427 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:37:56.670 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:37:56.670 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 139 k 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 139 k 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 513 k 2024-04-09T21:37:56.671 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:37:56.729 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-09T21:37:56.849 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-18.2.2-1187.gd540ebac.el8.x86_64 781 kB/s | 139 kB 00:00 2024-04-09T21:37:56.850 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:37:56.850 INFO:teuthology.orchestra.run.smithi137.stdout:Total 775 kB/s | 139 kB 00:00 2024-04-09T21:37:56.850 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:37:56.856 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:37:56.856 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:37:56.911 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:37:56.920 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:37:56.973 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-09T21:37:57.038 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:37:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:37:57.391 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-09T21:37:57.391 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 1/2 2024-04-09T21:37:57.534 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-09T21:38:07.233 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:38:07.935 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:07.935 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:38:07.943 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:07.943 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:38:07.943 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-09T21:38:07.943 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-09T21:38:07.943 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:07.943 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:38:08.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-04-09T21:38:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:38:08.492 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:38:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:38:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:38:08.492 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:08.503 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:08.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-04-09T21:38:09.018 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:10.325 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-09T21:38:10.714 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-09T21:38:10.893 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 kB/s | 4.1 kB 00:00 2024-04-09T21:38:11.176 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 kB/s | 4.5 kB 00:00 2024-04-09T21:38:13.357 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:13.484 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 139 k 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:38:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:38:13.486 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:13.486 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 139 k 2024-04-09T21:38:13.486 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 513 k 2024-04-09T21:38:13.486 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:38:13.551 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:13.552 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:38:13.553 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:13.668 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-18.2.2-1187.gd540ebac.el8.x86_64 764 kB/s | 139 kB 00:00 2024-04-09T21:38:13.668 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:38:13.669 INFO:teuthology.orchestra.run.smithi063.stdout:Total 759 kB/s | 139 kB 00:00 2024-04-09T21:38:13.669 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:38:13.675 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:38:13.675 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:38:13.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-04-09T21:38:13.735 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:38:13.744 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:38:13.876 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:38:14.053 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:14.068 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:14.333 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:15.830 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-09T21:38:16.207 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-09T21:38:18.561 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:18.562 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:38:18.562 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 31 k 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 k 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 139 k 2024-04-09T21:38:18.563 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:38:18.730 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el8.x86_6 185 kB/s | 31 kB 00:00 2024-04-09T21:38:18.731 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:38:18.731 INFO:teuthology.orchestra.run.smithi137.stdout:Total 184 kB/s | 31 kB 00:00 2024-04-09T21:38:18.731 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:38:18.735 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:38:18.735 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:38:18.747 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:38:18.756 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:38:18.812 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:38:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:19.225 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:25.240 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:25.240 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:38:25.839 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:25.840 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:38:25.840 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:38:25.840 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:25.840 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:38:26.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-04-09T21:38:26.433 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:28.235 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 75 kB/s | 4.1 kB 00:00 2024-04-09T21:38:28.627 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-09T21:38:29.432 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:29.432 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:38:29.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:38:29.997 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:38:29.997 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:38:29.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:38:29.997 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:30.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-04-09T21:38:30.518 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:31.199 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:31.401 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:38:31.403 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:38:31.403 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:38:31.548 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-04-09T21:38:31.927 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:32.368 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 kB/s | 4.1 kB 00:00 2024-04-09T21:38:32.693 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-09T21:38:33.697 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2024-04-09T21:38:34.106 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-09T21:38:34.855 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:35.044 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:35.045 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:38:35.045 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:35.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-04-09T21:38:35.546 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:36.908 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:38:36.909 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:38:36.909 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-04-09T21:38:36.909 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:38:36.909 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 31 k 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 31 k 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 139 k 2024-04-09T21:38:36.910 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:38:37.048 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el8.x86_6 224 kB/s | 31 kB 00:00 2024-04-09T21:38:37.049 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:38:37.049 INFO:teuthology.orchestra.run.smithi063.stdout:Total 222 kB/s | 31 kB 00:00 2024-04-09T21:38:37.049 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:38:37.053 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:38:37.053 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:38:37.066 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:38:37.075 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:38:37.139 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:38:37.331 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:37.374 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-09T21:38:37.588 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:37.762 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-09T21:38:39.934 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:40.126 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:38:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:40.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-04-09T21:38:40.629 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:42.666 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 13 kB/s | 4.1 kB 00:00 2024-04-09T21:38:42.969 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 kB/s | 4.5 kB 00:00 2024-04-09T21:38:45.154 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:45.347 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:45.348 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:38:45.348 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:45.352 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:38:45.353 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:38:45.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-04-09T21:38:45.853 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:45.949 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:45.950 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:38:45.950 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:38:45.950 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:38:45.950 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:38:46.122 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-04-09T21:38:46.503 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:47.689 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 kB/s | 4.1 kB 00:00 2024-04-09T21:38:48.091 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-09T21:38:48.340 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 22 kB/s | 4.1 kB 00:00 2024-04-09T21:38:48.821 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 21 kB/s | 4.5 kB 00:00 2024-04-09T21:38:50.303 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:50.493 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:50.494 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:38:50.494 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:50.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-04-09T21:38:51.006 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:51.363 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:51.577 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:38:51.578 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:38:51.578 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:38:51.739 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-04-09T21:38:52.120 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:52.863 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-09T21:38:53.183 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-09T21:38:53.991 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:38:54.395 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-09T21:38:55.359 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:55.553 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:38:55.554 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:38:55.554 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:38:55.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-04-09T21:38:56.069 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:57.010 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:38:57.225 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:38:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:38:57.226 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:38:57.384 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-04-09T21:38:57.765 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:38:58.006 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-09T21:38:58.382 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-09T21:38:59.632 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-09T21:38:59.985 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 kB/s | 4.5 kB 00:00 2024-04-09T21:39:00.578 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:39:00.770 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:39:00.771 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-04-09T21:39:00.772 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:39:00.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-04-09T21:39:01.280 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:02.564 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:39:02.764 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:02.765 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:39:02.765 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:39:02.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-04-09T21:39:03.125 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-09T21:39:03.292 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:03.472 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-09T21:39:04.909 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 72 kB/s | 4.1 kB 00:00 2024-04-09T21:39:05.292 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-09T21:39:05.861 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:39:05.861 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:05.861 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 95 k 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:39:05.862 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:05.863 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 95 k 2024-04-09T21:39:05.863 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 280 k 2024-04-09T21:39:05.863 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:39:06.052 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el8.x86_64.rpm 503 kB/s | 95 kB 00:00 2024-04-09T21:39:06.052 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:39:06.052 INFO:teuthology.orchestra.run.smithi137.stdout:Total 500 kB/s | 95 kB 00:00 2024-04-09T21:39:06.052 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:39:06.063 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:39:06.063 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:39:06.119 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:39:06.127 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:39:06.284 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:39:06.483 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:06.750 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:07.871 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:39:08.080 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:08.081 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:39:08.081 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:39:08.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-04-09T21:39:08.607 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:10.564 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-09T21:39:10.965 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-09T21:39:13.519 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:39:13.723 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:13.725 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:39:13.725 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:39:13.876 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-04-09T21:39:14.096 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:14.097 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:39:14.260 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:14.660 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:14.661 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:39:14.661 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:39:14.661 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:14.661 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:39:14.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-04-09T21:39:15.183 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:16.091 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-09T21:39:16.491 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-09T21:39:17.115 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-09T21:39:17.460 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-09T21:39:19.042 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-09T21:39:19.247 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:19.248 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-09T21:39:19.248 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:39:19.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-04-09T21:39:19.780 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:19.829 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.5 M 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:39:19.830 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:19.831 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:39:19.831 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:19.831 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.5 M 2024-04-09T21:39:19.831 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 16 M 2024-04-09T21:39:19.831 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:39:21.213 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el8.x86_64.rpm 2.5 MB/s | 3.5 MB 00:01 2024-04-09T21:39:21.213 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:39:21.214 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.5 MB/s | 3.5 MB 00:01 2024-04-09T21:39:21.214 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:39:21.228 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:39:21.228 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:39:21.301 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:39:21.310 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:39:21.696 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-09T21:39:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:39:21.916 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:22.117 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 kB/s | 4.5 kB 00:00 2024-04-09T21:39:22.319 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:24.905 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 95 k 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 95 k 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 280 k 2024-04-09T21:39:24.906 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:39:25.096 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el8.x86_64.rpm 501 kB/s | 95 kB 00:00 2024-04-09T21:39:25.097 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:39:25.097 INFO:teuthology.orchestra.run.smithi063.stdout:Total 498 kB/s | 95 kB 00:00 2024-04-09T21:39:25.097 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:39:25.109 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:39:25.109 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:39:25.171 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:39:25.180 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:39:25.342 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:39:25.536 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:25.811 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:30.387 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:30.388 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:39:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:30.937 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:39:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:39:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:30.937 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:39:31.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-04-09T21:39:31.458 INFO:teuthology.orchestra.run.smithi137.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:33.398 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-09T21:39:33.746 INFO:teuthology.orchestra.run.smithi137.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-09T21:39:35.579 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:35.580 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:39:36.145 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-04-09T21:39:36.145 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:36.145 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-04-09T21:39:36.145 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:36.145 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-04-09T21:39:36.146 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 181 k 2024-04-09T21:39:36.146 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:36.146 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-04-09T21:39:36.146 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-04-09T21:39:36.146 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-04-09T21:39:36.146 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:36.147 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 181 k 2024-04-09T21:39:36.147 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 572 k 2024-04-09T21:39:36.147 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-04-09T21:39:36.169 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:36.169 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:39:36.169 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:39:36.169 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:36.169 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:39:36.335 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-04-09T21:39:36.541 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el8.x86_64.rpm 460 kB/s | 181 kB 00:00 2024-04-09T21:39:36.542 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:39:36.542 INFO:teuthology.orchestra.run.smithi137.stdout:Total 458 kB/s | 181 kB 00:00 2024-04-09T21:39:36.542 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-04-09T21:39:36.554 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-04-09T21:39:36.554 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-04-09T21:39:36.611 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-04-09T21:39:36.619 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-04-09T21:39:36.716 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:36.738 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-04-09T21:39:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:38.422 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 28 kB/s | 4.1 kB 00:00 2024-04-09T21:39:38.868 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-09T21:39:41.716 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.5 M 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.5 M 2024-04-09T21:39:41.717 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 16 M 2024-04-09T21:39:41.718 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:39:42.367 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el8.x86_64.rpm 5.3 MB/s | 3.5 MB 00:00 2024-04-09T21:39:42.367 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:39:42.368 INFO:teuthology.orchestra.run.smithi063.stdout:Total 5.3 MB/s | 3.5 MB 00:00 2024-04-09T21:39:42.368 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:39:42.382 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:39:42.382 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:39:42.456 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:39:42.465 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:39:42.940 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:39:43.094 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:43.532 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:45.796 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:45.796 INFO:teuthology.orchestra.run.smithi137.stdout:Installed products updated. 2024-04-09T21:39:46.358 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:46.358 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-04-09T21:39:46.358 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:39:46.358 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:39:46.358 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-04-09T21:39:46.539 DEBUG:teuthology.parallel:result is None 2024-04-09T21:39:51.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:51.589 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:39:52.180 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:52.181 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:39:52.181 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:39:52.181 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:52.181 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:39:52.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-04-09T21:39:52.722 INFO:teuthology.orchestra.run.smithi063.stdout:Updating Subscription Management repositories. 2024-04-09T21:39:54.564 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-09T21:39:54.935 INFO:teuthology.orchestra.run.smithi063.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-09T21:39:57.654 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-09T21:39:57.654 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 181 k 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 181 k 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 572 k 2024-04-09T21:39:57.655 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-09T21:39:57.838 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el8.x86_64.rpm 996 kB/s | 181 kB 00:00 2024-04-09T21:39:57.838 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-09T21:39:57.839 INFO:teuthology.orchestra.run.smithi063.stdout:Total 989 kB/s | 181 kB 00:00 2024-04-09T21:39:57.839 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-09T21:39:57.852 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-09T21:39:57.852 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-09T21:39:57.912 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-09T21:39:57.921 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-09T21:39:58.038 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-09T21:39:58.232 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:39:58.510 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:40:09.161 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-09T21:40:09.161 INFO:teuthology.orchestra.run.smithi063.stdout:Installed products updated. 2024-04-09T21:40:09.748 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:09.748 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-09T21:40:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-09T21:40:09.749 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:09.749 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-09T21:40:09.923 DEBUG:teuthology.parallel:result is None 2024-04-09T21:40:09.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-09T21:40:10.078 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-09T21:40:10.100 INFO:teuthology.orchestra.run.smithi063.stdout:18.2.2-1187.gd540ebac.el8 2024-04-09T21:40:10.101 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el8 2024-04-09T21:40:10.101 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-09T21:40:10.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-09T21:40:10.255 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-09T21:40:10.283 INFO:teuthology.orchestra.run.smithi137.stdout:18.2.2-1187.gd540ebac.el8 2024-04-09T21:40:10.283 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el8 2024-04-09T21:40:10.283 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-09T21:40:10.285 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-09T21:40:10.285 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:10.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-09T21:40:10.315 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:10.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-09T21:40:10.357 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-09T21:40:10.357 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:10.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-04-09T21:40:10.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-09T21:40:10.453 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:10.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-04-09T21:40:10.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-09T21:40:10.553 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-09T21:40:10.553 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:10.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-09T21:40:10.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-09T21:40:10.649 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:10.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-09T21:40:10.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-09T21:40:10.746 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-09T21:40:10.747 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:10.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-04-09T21:40:10.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-09T21:40:10.843 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:10.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-04-09T21:40:10.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-09T21:40:10.939 INFO:teuthology.run_tasks:Running task ceph... 2024-04-09T21:40:11.029 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-09T21:40:11.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-04-09T21:40:11.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-04-09T21:40:11.059 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-09T21:40:11.059 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-09T21:40:11.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-09T21:40:11.127 INFO:tasks.ceph:Creating extra log directories... 2024-04-09T21:40:11.127 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-09T21:40:11.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-09T21:40:11.204 INFO:tasks.ceph:Creating ceph cluster c1... 2024-04-09T21:40:11.204 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-09T21:40:11.204 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7647917', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6358, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'targets': {'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6JmiQup3jBbUc4fDfxA1Cc87n92MyW1E37W5w047Bu63ZvbHk+9o3MyQjRiYMiRUTXFQavy0sy2i0bbdnktEc=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/q3UvERIx3e325Ay3goK8EiCuOC295xGHManCHbnvw67Gvt8s2Nmjbhmw1zIntD9LFijFSyuVpPD5AtVlfbmU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-09_01:21:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-09T21:40:11.205 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-09T21:40:11.235 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-09T21:40:11.253 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-04-09T21:40:11.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-04-09T21:40:11.320 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:11.320 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-04-09T21:40:11.376 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-09T21:40:11.377 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 180455 Links: 1 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-09 21:39:43.280043482 +0000 2024-04-09T21:40:11.435 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-09 21:32:02.944487291 +0000 2024-04-09T21:40:11.436 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-09 21:32:02.944487291 +0000 2024-04-09T21:40:11.436 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-09T21:40:11.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-09T21:40:11.502 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-09T21:40:11.502 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-09T21:40:11.502 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000201498 s, 2.5 MB/s 2024-04-09T21:40:11.503 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-09T21:40:11.561 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 183518 Links: 1 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-09 21:39:43.280043482 +0000 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-09 21:32:03.283484112 +0000 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-09 21:32:03.283484112 +0000 2024-04-09T21:40:11.620 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-09T21:40:11.620 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-09T21:40:11.686 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-09T21:40:11.686 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-09T21:40:11.686 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000221509 s, 2.3 MB/s 2024-04-09T21:40:11.687 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-09T21:40:11.745 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 175676 Links: 1 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-09 21:39:43.281043473 +0000 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-09 21:32:03.625480904 +0000 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-09 21:32:03.625480904 +0000 2024-04-09T21:40:11.803 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-09T21:40:11.804 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-09T21:40:11.869 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-09T21:40:11.869 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-09T21:40:11.870 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000179058 s, 2.9 MB/s 2024-04-09T21:40:11.871 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-09T21:40:11.930 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 182559 Links: 1 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-09 21:39:43.281043473 +0000 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-09 21:32:04.013477265 +0000 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-09 21:32:04.013477265 +0000 2024-04-09T21:40:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-09T21:40:11.989 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-09T21:40:12.055 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-09T21:40:12.055 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-09T21:40:12.055 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000229972 s, 2.2 MB/s 2024-04-09T21:40:12.056 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-09T21:40:12.114 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:12.114 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-09T21:40:12.114 INFO:tasks.ceph:Generating config... 2024-04-09T21:40:12.116 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-09T21:40:12.116 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-09T21:40:12.116 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-09T21:40:12.116 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-09T21:40:12.117 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-09T21:40:12.118 INFO:tasks.ceph:[client] setuser = ceph 2024-04-09T21:40:12.118 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-09T21:40:12.118 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-09T21:40:12.118 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-09T21:40:12.118 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-09T21:40:12.118 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-09T21:40:12.118 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-09T21:40:12.118 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-09T21:40:12.118 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-09T21:40:12.118 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-09T21:40:12.118 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-09T21:40:12.119 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-09T21:40:12.119 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-09T21:40:12.119 INFO:tasks.ceph:Setting up c1.mon.a... 2024-04-09T21:40:12.119 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-04-09T21:40:12.203 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/c1.keyring 2024-04-09T21:40:12.206 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-04-09T21:40:12.252 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-09T21:40:12.279 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.63')] 2024-04-09T21:40:12.280 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.63'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-09T21:40:12.280 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:12.280 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-09T21:40:12.336 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.63 --print /home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:12.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:40:12.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:40:12.419 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:12.419 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: generated fsid 1a5ffe85-7a30-48a1-8205-048631064543 2024-04-09T21:40:12.419 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = pacific 2024-04-09T21:40:12.419 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 0 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:fsid 1a5ffe85-7a30-48a1-8205-048631064543 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:last_changed 2024-04-09T21:40:12.417842+0000 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-04-09T21:40:12.417842+0000 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:min_mon_release 16 (pacific) 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:election_strategy: 1 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-09T21:40:12.420 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-04-09T21:40:12.421 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-09T21:40:12.479 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 1a5ffe85-7a30-48a1-8205-048631064543... 2024-04-09T21:40:12.480 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-09T21:40:12.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-09T21:40:12.566 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-04-09T21:40:12.567 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-04-09T21:40:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-04-09T21:40:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = 172.21.15.63 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 1a5ffe85-7a30-48a1-8205-048631064543 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-04-09T21:40:12.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-04-09T21:40:12.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: rocksdb delete range threshold = 0 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-04-09T21:40:12.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-09T21:40:12.573 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-04-09T21:40:12.574 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw = 20 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw sync = 20 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt require ssl = False 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms backend = testing 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw curl low speed time = 300 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw data log num shards = 4 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw md log max shards = 4 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync data inject err probability = 0.1 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync log trim interval = 0 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync meta inject err probability = 0.1 2024-04-09T21:40:12.575 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync obj etag verify = True 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout: setgroup = ceph 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout: setuser = ceph 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout: rgw data notify interval msec = 0 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_omap_entries_per_request = 10 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_omap_entries_per_request = 1000 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-04-09T21:40:12.576 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_omap_entries_per_request = 10000 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-04-09T21:40:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-04-09T21:40:12.578 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-04-09T21:40:12.579 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.63 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 1a5ffe85-7a30-48a1-8205-048631064543 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.580 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-04-09T21:40:12.581 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: rocksdb delete range threshold = 0 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.582 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-04-09T21:40:12.583 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-09T21:40:12.584 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 20 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw sync = 20 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt require ssl = False 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms backend = testing 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw curl low speed time = 300 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data log num shards = 4 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw md log max shards = 4 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync data inject err probability = 0.1 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync log trim interval = 0 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync meta inject err probability = 0.1 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync obj etag verify = True 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: setgroup = ceph 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout: setuser = ceph 2024-04-09T21:40:12.585 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data notify interval msec = 0 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 1000 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-04-09T21:40:12.586 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10000 2024-04-09T21:40:12.588 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-04-09T21:40:12.589 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-04-09T21:40:12.687 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-09T21:40:12.687 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:12.687 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-04-09T21:40:12.704 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:12.704 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-04-09T21:40:12.763 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-04-09T21:40:12.763 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:12.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/c1.keyring 2024-04-09T21:40:12.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-09T21:40:12.841 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:12.841 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:12.899 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-04-09T21:40:12.899 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:12.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/c1.keyring 2024-04-09T21:40:12.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-09T21:40:12.944 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:12.944 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:13.004 INFO:tasks.ceph:Setting up mon nodes... 2024-04-09T21:40:13.004 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-09T21:40:13.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-04-09T21:40:13.062 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-04-09T21:40:13.064 INFO:tasks.ceph:Setting up mds nodes... 2024-04-09T21:40:13.064 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-09T21:40:13.065 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-04-09T21:40:13.110 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/c1.client.0.keyring 2024-04-09T21:40:13.123 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-04-09T21:40:13.168 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/c1.client.1.keyring 2024-04-09T21:40:13.181 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-09T21:40:13.182 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-09T21:40:13.182 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-04-09T21:40:13.208 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:13.209 INFO:tasks.ceph:role: c1.osd.0 2024-04-09T21:40:13.209 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-09T21:40:13.209 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-09T21:40:13.284 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-09T21:40:13.285 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-09T21:40:13.719 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-04-09T21:40:13.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-04-09T21:40:13.721 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-04-09T21:40:13.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-04-09T21:40:13.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-04-09T21:40:14.005 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:14.005 INFO:tasks.ceph:role: c1.osd.1 2024-04-09T21:40:14.005 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-09T21:40:14.006 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-04-09T21:40:14.081 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-09T21:40:14.082 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-09T21:40:14.082 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-09T21:40:14.082 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-09T21:40:14.508 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-04-09T21:40:14.510 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-04-09T21:40:14.510 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-04-09T21:40:14.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-04-09T21:40:14.656 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-04-09T21:40:14.725 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:14.725 INFO:tasks.ceph:role: c1.osd.2 2024-04-09T21:40:14.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-09T21:40:14.725 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-09T21:40:14.801 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-09T21:40:14.802 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-09T21:40:14.802 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-09T21:40:15.239 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-04-09T21:40:15.240 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-04-09T21:40:15.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-04-09T21:40:15.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-04-09T21:40:15.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:15.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:15.522+0000 7fce4cbe47c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-04-09T21:40:15.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:15.525+0000 7fce4cbe47c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-04-09T21:40:15.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:15.525+0000 7fce4cbe47c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-09T21:40:15.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:15.525+0000 7fce4cbe47c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-09T21:40:15.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:15.525+0000 7fce4cbe47c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-09T21:40:15.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:15.530+0000 7fce4cbe47c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-04-09T21:40:17.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-04-09T21:40:17.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:17.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:17.741+0000 7fc5349f87c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-04-09T21:40:17.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:17.741+0000 7fc5349f87c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-04-09T21:40:17.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:17.741+0000 7fc5349f87c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-09T21:40:17.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:17.741+0000 7fc5349f87c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-09T21:40:17.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:17.741+0000 7fc5349f87c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-09T21:40:17.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:17.742+0000 7fc5349f87c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-04-09T21:40:19.827 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-04-09T21:40:19.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:19.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:19.952+0000 7ff5ec9d77c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-04-09T21:40:19.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:19.952+0000 7ff5ec9d77c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-04-09T21:40:19.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:19.952+0000 7ff5ec9d77c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-09T21:40:19.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:19.952+0000 7ff5ec9d77c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-09T21:40:19.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:19.952+0000 7ff5ec9d77c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-09T21:40:19.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:40:19.953+0000 7ff5ec9d77c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-04-09T21:40:22.036 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-04-09T21:40:22.069 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-09T21:40:22.070 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:22.070 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-04-09T21:40:22.141 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:22.141 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-04-09T21:40:22.211 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:22.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-04-09T21:40:22.281 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:22.281 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-04-09T21:40:22.352 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:22.352 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-04-09T21:40:22.411 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:22.411 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-04-09T21:40:22.469 INFO:tasks.ceph:Adding keys to all mons... 2024-04-09T21:40:22.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/c1.keyring 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQC9tRVmVDafAxAAwcpSyNlxMW/nxijVczSVlw== 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQC/tRVm30k8HxAA/zFTrPhZtO4IIAN1v5ro4g== 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDBtRVm3fJCLBAAkhgCW3lRmfgEp2rouuRC8Q== 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-04-09T21:40:22.537 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDDtRVmKx3VOBAAEBdDOVTb1qLMALoTe1e9zA== 2024-04-09T21:40:22.538 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-04-09T21:40:22.538 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQC9tRVmGS98BhAA8mIlh1B37gGOrs0LxJNQyw== 2024-04-09T21:40:22.538 INFO:teuthology.orchestra.run.smithi063.stdout:[client.1] 2024-04-09T21:40:22.538 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQC9tRVm9+7yCRAAvEOQP3EeXHJfel1RUe3Iew== 2024-04-09T21:40:22.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-09T21:40:22.636 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-09T21:40:22.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-09T21:40:22.746 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-09T21:40:22.802 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-09T21:40:22.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-09T21:40:22.919 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-09T21:40:22.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-04-09T21:40:22.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-04-09T21:40:23.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-04-09T21:40:23.077 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-04-09T21:40:23.134 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-04-09T21:40:23.135 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-04-09T21:40:23.135 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-04-09T21:40:23.176 INFO:tasks.ceph.c1.mon.a:Started 2024-04-09T21:40:23.176 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-04-09T21:40:23.177 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-04-09T21:40:23.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-04-09T21:40:23.179 INFO:tasks.ceph.c1.mgr.x:Started 2024-04-09T21:40:23.179 INFO:tasks.ceph:Setting crush tunables to default 2024-04-09T21:40:23.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 osd crush tunables default 2024-04-09T21:40:23.398 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-04-09T21:40:23.409 INFO:tasks.ceph:check_enable_crimson: False 2024-04-09T21:40:23.409 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-04-09T21:40:23.410 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:23.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-04-09T21:40:23.438 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:23.438 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-04-09T21:40:23.506 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:23.506 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-04-09T21:40:23.574 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 osd new 1494dd05-b83d-494a-8948-52e885c3c67c 0 2024-04-09T21:40:23.834 INFO:teuthology.orchestra.run.smithi063.stdout:0 2024-04-09T21:40:23.847 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 osd new a2f1f221-aa17-4fb7-83dc-9a9d7950f431 1 2024-04-09T21:40:23.887 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:23.885+0000 7fb7beaf6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-09T21:40:23.972 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:23.970+0000 7fb7beaf6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-09T21:40:24.050 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.048+0000 7fb7beaf6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-09T21:40:24.068 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-04-09T21:40:24.080 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 osd new eca9bdcb-53db-472b-9d01-7e3178499e87 2 2024-04-09T21:40:24.125 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.123+0000 7fb7beaf6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-09T21:40:24.311 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.308+0000 7fb7beaf6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-09T21:40:24.343 INFO:teuthology.orchestra.run.smithi063.stdout:2 2024-04-09T21:40:24.356 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-04-09T21:40:24.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-04-09T21:40:24.399 INFO:tasks.ceph.c1.osd.0:Started 2024-04-09T21:40:24.399 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-04-09T21:40:24.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-04-09T21:40:24.401 INFO:tasks.ceph.c1.osd.1:Started 2024-04-09T21:40:24.402 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-04-09T21:40:24.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-04-09T21:40:24.404 INFO:tasks.ceph.c1.osd.2:Started 2024-04-09T21:40:24.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-09T21:40:24.522 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.519+0000 7fb7beaf6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-09T21:40:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:24.634 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":5,"fsid":"1a5ffe85-7a30-48a1-8205-048631064543","created":"2024-04-09T21:40:23.296971+0000","modified":"2024-04-09T21:40:24.339168+0000","last_up_change":"0.000000","last_in_change":"2024-04-09T21:40:24.339168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1494dd05-b83d-494a-8948-52e885c3c67c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2f1f221-aa17-4fb7-83dc-9a9d7950f431","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eca9bdcb-53db-472b-9d01-7e3178499e87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:24.638 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.635+0000 7fb7beaf6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-09T21:40:24.646 INFO:tasks.ceph.ceph_manager.c1:[] 2024-04-09T21:40:24.647 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-09T21:40:24.738 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.736+0000 7fb7beaf6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-09T21:40:24.818 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:24.816+0000 7fb7beaf6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-09T21:40:24.949 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-09T21:40:25.017 INFO:tasks.ceph.c1.osd.2.smithi063.stderr:2024-04-09T21:40:25.015+0000 7f9de62bc7c0 -1 Falling back to public interface 2024-04-09T21:40:25.020 INFO:tasks.ceph.c1.osd.0.smithi063.stderr:2024-04-09T21:40:25.017+0000 7fe90ca997c0 -1 Falling back to public interface 2024-04-09T21:40:25.039 INFO:tasks.ceph.c1.osd.1.smithi063.stderr:2024-04-09T21:40:25.037+0000 7f2e443167c0 -1 Falling back to public interface 2024-04-09T21:40:25.062 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:25.060+0000 7fb7beaf6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-09T21:40:25.139 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:25.136+0000 7fb7beaf6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-09T21:40:25.182 INFO:teuthology.misc.health.smithi063.stdout: 2024-04-09T21:40:25.182 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":5,"fsid":"1a5ffe85-7a30-48a1-8205-048631064543","created":"2024-04-09T21:40:23.296971+0000","modified":"2024-04-09T21:40:24.339168+0000","last_up_change":"0.000000","last_in_change":"2024-04-09T21:40:24.339168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1494dd05-b83d-494a-8948-52e885c3c67c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2f1f221-aa17-4fb7-83dc-9a9d7950f431","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eca9bdcb-53db-472b-9d01-7e3178499e87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:25.194 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-09T21:40:25.547 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:25.544+0000 7fb7beaf6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-09T21:40:25.881 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:25.878+0000 7fb7beaf6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-09T21:40:26.048 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:26.045+0000 7fb7beaf6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-09T21:40:26.130 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:26.128+0000 7fb7beaf6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-09T21:40:26.879 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:26.876+0000 7fb7beaf6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-09T21:40:26.965 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:26.963+0000 7fb7beaf6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-09T21:40:27.104 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:27.101+0000 7fb7beaf6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-09T21:40:27.682 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:27.679+0000 7fb7beaf6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-09T21:40:27.735 INFO:tasks.ceph.c1.osd.0.smithi063.stderr:2024-04-09T21:40:27.732+0000 7fe90ca997c0 -1 osd.0 0 log_to_monitors true 2024-04-09T21:40:27.748 INFO:tasks.ceph.c1.osd.1.smithi063.stderr:2024-04-09T21:40:27.745+0000 7f2e443167c0 -1 osd.1 0 log_to_monitors true 2024-04-09T21:40:27.803 INFO:tasks.ceph.c1.osd.2.smithi063.stderr:2024-04-09T21:40:27.801+0000 7f9de62bc7c0 -1 osd.2 0 log_to_monitors true 2024-04-09T21:40:27.864 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:27.862+0000 7fb7beaf6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-09T21:40:28.037 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:28.035+0000 7fb7beaf6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-09T21:40:28.925 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:28.923+0000 7fb7beaf6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-09T21:40:29.184 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:29.182+0000 7fb7beaf6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-09T21:40:29.270 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:29.267+0000 7fb7beaf6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-09T21:40:29.683 INFO:tasks.ceph.c1.osd.1.smithi063.stderr:2024-04-09T21:40:29.680+0000 7f2e386ce700 -1 osd.1 0 waiting for initial osdmap 2024-04-09T21:40:29.692 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:29.690+0000 7fb7beaf6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-09T21:40:29.695 INFO:tasks.ceph.c1.osd.1.smithi063.stderr:2024-04-09T21:40:29.693+0000 7f2e302a8700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-09T21:40:29.712 INFO:tasks.ceph.c1.osd.0.smithi063.stderr:2024-04-09T21:40:29.710+0000 7fe900e51700 -1 osd.0 0 waiting for initial osdmap 2024-04-09T21:40:29.728 INFO:tasks.ceph.c1.osd.0.smithi063.stderr:2024-04-09T21:40:29.726+0000 7fe8f8a2b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-09T21:40:30.312 INFO:tasks.ceph.c1.osd.2.smithi063.stderr:2024-04-09T21:40:30.309+0000 7f9dda674700 -1 osd.2 0 waiting for initial osdmap 2024-04-09T21:40:30.326 INFO:tasks.ceph.c1.osd.2.smithi063.stderr:2024-04-09T21:40:30.324+0000 7f9dd224e700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-09T21:40:30.723 INFO:tasks.ceph.c1.mgr.x.smithi063.stderr:2024-04-09T21:40:30.721+0000 7fb779d70700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-09T21:40:31.497 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-09T21:40:31.843 INFO:teuthology.misc.health.smithi063.stdout: 2024-04-09T21:40:31.843 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":9,"fsid":"1a5ffe85-7a30-48a1-8205-048631064543","created":"2024-04-09T21:40:23.296971+0000","modified":"2024-04-09T21:40:31.313733+0000","last_up_change":"2024-04-09T21:40:31.313733+0000","last_in_change":"2024-04-09T21:40:24.339168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1494dd05-b83d-494a-8948-52e885c3c67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6801","nonce":40866},{"type":"v1","addr":"172.21.15.63:6802","nonce":40866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6803","nonce":40866},{"type":"v1","addr":"172.21.15.63:6804","nonce":40866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6807","nonce":40866},{"type":"v1","addr":"172.21.15.63:6808","nonce":40866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6805","nonce":40866},{"type":"v1","addr":"172.21.15.63:6806","nonce":40866}]},"public_addr":"172.21.15.63:6802/40866","cluster_addr":"172.21.15.63:6804/40866","heartbeat_back_addr":"172.21.15.63:6808/40866","heartbeat_front_addr":"172.21.15.63:6806/40866","state":["exists","up"]},{"osd":1,"uuid":"a2f1f221-aa17-4fb7-83dc-9a9d7950f431","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":40879},{"type":"v1","addr":"172.21.15.63:6810","nonce":40879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6811","nonce":40879},{"type":"v1","addr":"172.21.15.63:6812","nonce":40879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6815","nonce":40879},{"type":"v1","addr":"172.21.15.63:6816","nonce":40879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6813","nonce":40879},{"type":"v1","addr":"172.21.15.63:6814","nonce":40879}]},"public_addr":"172.21.15.63:6810/40879","cluster_addr":"172.21.15.63:6812/40879","heartbeat_back_addr":"172.21.15.63:6816/40879","heartbeat_front_addr":"172.21.15.63:6814/40879","state":["exists","up"]},{"osd":2,"uuid":"eca9bdcb-53db-472b-9d01-7e3178499e87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":40867},{"type":"v1","addr":"172.21.15.63:6817","nonce":40867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":40867},{"type":"v1","addr":"172.21.15.63:6819","nonce":40867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":40867},{"type":"v1","addr":"172.21.15.63:6823","nonce":40867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":40867},{"type":"v1","addr":"172.21.15.63:6821","nonce":40867}]},"public_addr":"172.21.15.63:6817/40867","cluster_addr":"172.21.15.63:6819/40867","heartbeat_back_addr":"172.21.15.63:6823/40867","heartbeat_front_addr":"172.21.15.63:6821/40867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.718569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.709345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:31.856 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-09T21:40:31.856 INFO:tasks.ceph:Creating RBD pool 2024-04-09T21:40:31.856 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 osd pool create rbd 8 2024-04-09T21:40:32.329 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'rbd' created 2024-04-09T21:40:32.340 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-09T21:40:33.335 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-09T21:40:33.347 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-04-09T21:40:33.348 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-04-09T21:40:33.348 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-04-09T21:40:33.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-04-09T21:40:33.349 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-09T21:40:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:33.796 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":40484},{"type":"v1","addr":"172.21.15.63:6825","nonce":40484}]},"active_addr":"172.21.15.63:6825/40484","active_change":"2024-04-09T21:40:29.703892+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3087896608}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2347843667}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3580912050}]}]} 2024-04-09T21:40:33.799 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-04-09T21:40:33.799 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-04-09T21:40:33.799 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-09T21:40:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:34.148 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":11,"fsid":"1a5ffe85-7a30-48a1-8205-048631064543","created":"2024-04-09T21:40:23.296971+0000","modified":"2024-04-09T21:40:33.320776+0000","last_up_change":"2024-04-09T21:40:31.313733+0000","last_in_change":"2024-04-09T21:40:24.339168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-09T21:40:31.761760+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-09T21:40:32.222999+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1494dd05-b83d-494a-8948-52e885c3c67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6801","nonce":40866},{"type":"v1","addr":"172.21.15.63:6802","nonce":40866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6803","nonce":40866},{"type":"v1","addr":"172.21.15.63:6804","nonce":40866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6807","nonce":40866},{"type":"v1","addr":"172.21.15.63:6808","nonce":40866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6805","nonce":40866},{"type":"v1","addr":"172.21.15.63:6806","nonce":40866}]},"public_addr":"172.21.15.63:6802/40866","cluster_addr":"172.21.15.63:6804/40866","heartbeat_back_addr":"172.21.15.63:6808/40866","heartbeat_front_addr":"172.21.15.63:6806/40866","state":["exists","up"]},{"osd":1,"uuid":"a2f1f221-aa17-4fb7-83dc-9a9d7950f431","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":40879},{"type":"v1","addr":"172.21.15.63:6810","nonce":40879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6811","nonce":40879},{"type":"v1","addr":"172.21.15.63:6812","nonce":40879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6815","nonce":40879},{"type":"v1","addr":"172.21.15.63:6816","nonce":40879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6813","nonce":40879},{"type":"v1","addr":"172.21.15.63:6814","nonce":40879}]},"public_addr":"172.21.15.63:6810/40879","cluster_addr":"172.21.15.63:6812/40879","heartbeat_back_addr":"172.21.15.63:6816/40879","heartbeat_front_addr":"172.21.15.63:6814/40879","state":["exists","up"]},{"osd":2,"uuid":"eca9bdcb-53db-472b-9d01-7e3178499e87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":40867},{"type":"v1","addr":"172.21.15.63:6817","nonce":40867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":40867},{"type":"v1","addr":"172.21.15.63:6819","nonce":40867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":40867},{"type":"v1","addr":"172.21.15.63:6823","nonce":40867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":40867},{"type":"v1","addr":"172.21.15.63:6821","nonce":40867}]},"public_addr":"172.21.15.63:6817/40867","cluster_addr":"172.21.15.63:6819/40867","heartbeat_back_addr":"172.21.15.63:6823/40867","heartbeat_front_addr":"172.21.15.63:6821/40867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.718569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.709345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.793302+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:34.162 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-04-09T21:40:34.163 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-09T21:40:34.505 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:34.506 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":12,"fsid":"1a5ffe85-7a30-48a1-8205-048631064543","created":"2024-04-09T21:40:23.296971+0000","modified":"2024-04-09T21:40:34.325662+0000","last_up_change":"2024-04-09T21:40:31.313733+0000","last_in_change":"2024-04-09T21:40:24.339168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-09T21:40:31.761760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-09T21:40:32.222999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1494dd05-b83d-494a-8948-52e885c3c67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6801","nonce":40866},{"type":"v1","addr":"172.21.15.63:6802","nonce":40866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6803","nonce":40866},{"type":"v1","addr":"172.21.15.63:6804","nonce":40866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6807","nonce":40866},{"type":"v1","addr":"172.21.15.63:6808","nonce":40866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6805","nonce":40866},{"type":"v1","addr":"172.21.15.63:6806","nonce":40866}]},"public_addr":"172.21.15.63:6802/40866","cluster_addr":"172.21.15.63:6804/40866","heartbeat_back_addr":"172.21.15.63:6808/40866","heartbeat_front_addr":"172.21.15.63:6806/40866","state":["exists","up"]},{"osd":1,"uuid":"a2f1f221-aa17-4fb7-83dc-9a9d7950f431","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":40879},{"type":"v1","addr":"172.21.15.63:6810","nonce":40879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6811","nonce":40879},{"type":"v1","addr":"172.21.15.63:6812","nonce":40879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6815","nonce":40879},{"type":"v1","addr":"172.21.15.63:6816","nonce":40879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6813","nonce":40879},{"type":"v1","addr":"172.21.15.63:6814","nonce":40879}]},"public_addr":"172.21.15.63:6810/40879","cluster_addr":"172.21.15.63:6812/40879","heartbeat_back_addr":"172.21.15.63:6816/40879","heartbeat_front_addr":"172.21.15.63:6814/40879","state":["exists","up"]},{"osd":2,"uuid":"eca9bdcb-53db-472b-9d01-7e3178499e87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":40867},{"type":"v1","addr":"172.21.15.63:6817","nonce":40867}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":40867},{"type":"v1","addr":"172.21.15.63:6819","nonce":40867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":40867},{"type":"v1","addr":"172.21.15.63:6823","nonce":40867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":40867},{"type":"v1","addr":"172.21.15.63:6821","nonce":40867}]},"public_addr":"172.21.15.63:6817/40867","cluster_addr":"172.21.15.63:6819/40867","heartbeat_back_addr":"172.21.15.63:6823/40867","heartbeat_front_addr":"172.21.15.63:6821/40867","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.718569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.709345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:28.793302+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:34.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-04-09T21:40:34.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-04-09T21:40:34.521 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-04-09T21:40:34.689 INFO:teuthology.orchestra.run.smithi063.stdout:34359738370 2024-04-09T21:40:34.689 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-09T21:40:34.712 INFO:teuthology.orchestra.run.smithi063.stdout:34359738370 2024-04-09T21:40:34.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-09T21:40:34.713 INFO:teuthology.orchestra.run.smithi063.stdout:38654705666 2024-04-09T21:40:34.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-09T21:40:35.051 INFO:teuthology.orchestra.run.smithi063.stdout:34359738369 2024-04-09T21:40:35.065 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2024-04-09T21:40:35.106 INFO:teuthology.orchestra.run.smithi063.stdout:38654705665 2024-04-09T21:40:35.113 INFO:teuthology.orchestra.run.smithi063.stdout:34359738369 2024-04-09T21:40:35.119 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.2 2024-04-09T21:40:35.128 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2024-04-09T21:40:36.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-09T21:40:36.121 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-09T21:40:36.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-09T21:40:36.457 INFO:teuthology.orchestra.run.smithi063.stdout:34359738370 2024-04-09T21:40:36.477 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2024-04-09T21:40:36.477 DEBUG:teuthology.parallel:result is None 2024-04-09T21:40:36.516 INFO:teuthology.orchestra.run.smithi063.stdout:38654705666 2024-04-09T21:40:36.524 INFO:teuthology.orchestra.run.smithi063.stdout:34359738370 2024-04-09T21:40:36.531 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.2 2024-04-09T21:40:36.531 DEBUG:teuthology.parallel:result is None 2024-04-09T21:40:36.538 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2024-04-09T21:40:36.538 DEBUG:teuthology.parallel:result is None 2024-04-09T21:40:36.539 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-09T21:40:36.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-09T21:40:36.914 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:36.916 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-09T21:40:36.928 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-09T21:40:35.709067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83268,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489532,"statfs":{"total":322122547200,"available":322037280768,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337422+0000","last_change":"2024-04-09T21:40:33.331168+0000","last_active":"2024-04-09T21:40:33.337422+0000","last_peered":"2024-04-09T21:40:33.337422+0000","last_clean":"2024-04-09T21:40:33.337422+0000","last_became_active":"2024-04-09T21:40:33.330968+0000","last_became_peered":"2024-04-09T21:40:33.330968+0000","last_unstale":"2024-04-09T21:40:33.337422+0000","last_undegraded":"2024-04-09T21:40:33.337422+0000","last_fullsized":"2024-04-09T21:40:33.337422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T00:45:17.758251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337690+0000","last_change":"2024-04-09T21:40:33.332980+0000","last_active":"2024-04-09T21:40:33.337690+0000","last_peered":"2024-04-09T21:40:33.337690+0000","last_clean":"2024-04-09T21:40:33.337690+0000","last_became_active":"2024-04-09T21:40:33.332712+0000","last_became_peered":"2024-04-09T21:40:33.332712+0000","last_unstale":"2024-04-09T21:40:33.337690+0000","last_undegraded":"2024-04-09T21:40:33.337690+0000","last_fullsized":"2024-04-09T21:40:33.337690+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T04:44:28.541211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337357+0000","last_change":"2024-04-09T21:40:33.330316+0000","last_active":"2024-04-09T21:40:33.337357+0000","last_peered":"2024-04-09T21:40:33.337357+0000","last_clean":"2024-04-09T21:40:33.337357+0000","last_became_active":"2024-04-09T21:40:33.330002+0000","last_became_peered":"2024-04-09T21:40:33.330002+0000","last_unstale":"2024-04-09T21:40:33.337357+0000","last_undegraded":"2024-04-09T21:40:33.337357+0000","last_fullsized":"2024-04-09T21:40:33.337357+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T04:07:43.397005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-09T21:40:34.327957+0000","last_change":"2024-04-09T21:40:33.333516+0000","last_active":"2024-04-09T21:40:34.327957+0000","last_peered":"2024-04-09T21:40:34.327957+0000","last_clean":"2024-04-09T21:40:34.327957+0000","last_became_active":"2024-04-09T21:40:33.333299+0000","last_became_peered":"2024-04-09T21:40:33.333299+0000","last_unstale":"2024-04-09T21:40:34.327957+0000","last_undegraded":"2024-04-09T21:40:34.327957+0000","last_fullsized":"2024-04-09T21:40:34.327957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:20:00.792768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.924411+0000","last_change":"2024-04-09T21:40:33.330334+0000","last_active":"2024-04-09T21:40:33.924411+0000","last_peered":"2024-04-09T21:40:33.924411+0000","last_clean":"2024-04-09T21:40:33.924411+0000","last_became_active":"2024-04-09T21:40:33.330084+0000","last_became_peered":"2024-04-09T21:40:33.330084+0000","last_unstale":"2024-04-09T21:40:33.924411+0000","last_undegraded":"2024-04-09T21:40:33.924411+0000","last_fullsized":"2024-04-09T21:40:33.924411+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T22:37:02.882665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337336+0000","last_change":"2024-04-09T21:40:33.330248+0000","last_active":"2024-04-09T21:40:33.337336+0000","last_peered":"2024-04-09T21:40:33.337336+0000","last_clean":"2024-04-09T21:40:33.337336+0000","last_became_active":"2024-04-09T21:40:33.329921+0000","last_became_peered":"2024-04-09T21:40:33.329921+0000","last_unstale":"2024-04-09T21:40:33.337336+0000","last_undegraded":"2024-04-09T21:40:33.337336+0000","last_fullsized":"2024-04-09T21:40:33.337336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T23:29:06.943955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-09T21:40:34.327953+0000","last_change":"2024-04-09T21:40:33.333012+0000","last_active":"2024-04-09T21:40:34.327953+0000","last_peered":"2024-04-09T21:40:34.327953+0000","last_clean":"2024-04-09T21:40:34.327953+0000","last_became_active":"2024-04-09T21:40:33.332739+0000","last_became_peered":"2024-04-09T21:40:33.332739+0000","last_unstale":"2024-04-09T21:40:34.327953+0000","last_undegraded":"2024-04-09T21:40:34.327953+0000","last_fullsized":"2024-04-09T21:40:34.327953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T03:40:22.320370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337617+0000","last_change":"2024-04-09T21:40:33.332890+0000","last_active":"2024-04-09T21:40:33.337617+0000","last_peered":"2024-04-09T21:40:33.337617+0000","last_clean":"2024-04-09T21:40:33.337617+0000","last_became_active":"2024-04-09T21:40:33.332617+0000","last_became_peered":"2024-04-09T21:40:33.332617+0000","last_unstale":"2024-04-09T21:40:33.337617+0000","last_undegraded":"2024-04-09T21:40:33.337617+0000","last_fullsized":"2024-04-09T21:40:33.337617+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:57:01.831999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337676+0000","last_change":"2024-04-09T21:40:33.332883+0000","last_active":"2024-04-09T21:40:33.337676+0000","last_peered":"2024-04-09T21:40:33.337676+0000","last_clean":"2024-04-09T21:40:33.337676+0000","last_became_active":"2024-04-09T21:40:33.332620+0000","last_became_peered":"2024-04-09T21:40:33.332620+0000","last_unstale":"2024-04-09T21:40:33.337676+0000","last_undegraded":"2024-04-09T21:40:33.337676+0000","last_fullsized":"2024-04-09T21:40:33.337676+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T23:30:13.985736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28196,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829404,"statfs":{"total":107374182400,"available":107345309696,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27948,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829652,"statfs":{"total":107374182400,"available":107345563648,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-09T21:40:36.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-09T21:40:37.268 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:37.271 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-09T21:40:37.282 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-09T21:40:35.709067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83268,"kb_used_data":1448,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489532,"statfs":{"total":322122547200,"available":322037280768,"internally_reserved":0,"allocated":1482752,"data_stored":1256546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337422+0000","last_change":"2024-04-09T21:40:33.331168+0000","last_active":"2024-04-09T21:40:33.337422+0000","last_peered":"2024-04-09T21:40:33.337422+0000","last_clean":"2024-04-09T21:40:33.337422+0000","last_became_active":"2024-04-09T21:40:33.330968+0000","last_became_peered":"2024-04-09T21:40:33.330968+0000","last_unstale":"2024-04-09T21:40:33.337422+0000","last_undegraded":"2024-04-09T21:40:33.337422+0000","last_fullsized":"2024-04-09T21:40:33.337422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T00:45:17.758251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337690+0000","last_change":"2024-04-09T21:40:33.332980+0000","last_active":"2024-04-09T21:40:33.337690+0000","last_peered":"2024-04-09T21:40:33.337690+0000","last_clean":"2024-04-09T21:40:33.337690+0000","last_became_active":"2024-04-09T21:40:33.332712+0000","last_became_peered":"2024-04-09T21:40:33.332712+0000","last_unstale":"2024-04-09T21:40:33.337690+0000","last_undegraded":"2024-04-09T21:40:33.337690+0000","last_fullsized":"2024-04-09T21:40:33.337690+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T04:44:28.541211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337357+0000","last_change":"2024-04-09T21:40:33.330316+0000","last_active":"2024-04-09T21:40:33.337357+0000","last_peered":"2024-04-09T21:40:33.337357+0000","last_clean":"2024-04-09T21:40:33.337357+0000","last_became_active":"2024-04-09T21:40:33.330002+0000","last_became_peered":"2024-04-09T21:40:33.330002+0000","last_unstale":"2024-04-09T21:40:33.337357+0000","last_undegraded":"2024-04-09T21:40:33.337357+0000","last_fullsized":"2024-04-09T21:40:33.337357+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T04:07:43.397005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-09T21:40:34.327957+0000","last_change":"2024-04-09T21:40:33.333516+0000","last_active":"2024-04-09T21:40:34.327957+0000","last_peered":"2024-04-09T21:40:34.327957+0000","last_clean":"2024-04-09T21:40:34.327957+0000","last_became_active":"2024-04-09T21:40:33.333299+0000","last_became_peered":"2024-04-09T21:40:33.333299+0000","last_unstale":"2024-04-09T21:40:34.327957+0000","last_undegraded":"2024-04-09T21:40:34.327957+0000","last_fullsized":"2024-04-09T21:40:34.327957+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:20:00.792768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.924411+0000","last_change":"2024-04-09T21:40:33.330334+0000","last_active":"2024-04-09T21:40:33.924411+0000","last_peered":"2024-04-09T21:40:33.924411+0000","last_clean":"2024-04-09T21:40:33.924411+0000","last_became_active":"2024-04-09T21:40:33.330084+0000","last_became_peered":"2024-04-09T21:40:33.330084+0000","last_unstale":"2024-04-09T21:40:33.924411+0000","last_undegraded":"2024-04-09T21:40:33.924411+0000","last_fullsized":"2024-04-09T21:40:33.924411+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T22:37:02.882665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337336+0000","last_change":"2024-04-09T21:40:33.330248+0000","last_active":"2024-04-09T21:40:33.337336+0000","last_peered":"2024-04-09T21:40:33.337336+0000","last_clean":"2024-04-09T21:40:33.337336+0000","last_became_active":"2024-04-09T21:40:33.329921+0000","last_became_peered":"2024-04-09T21:40:33.329921+0000","last_unstale":"2024-04-09T21:40:33.337336+0000","last_undegraded":"2024-04-09T21:40:33.337336+0000","last_fullsized":"2024-04-09T21:40:33.337336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T23:29:06.943955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-09T21:40:34.327953+0000","last_change":"2024-04-09T21:40:33.333012+0000","last_active":"2024-04-09T21:40:34.327953+0000","last_peered":"2024-04-09T21:40:34.327953+0000","last_clean":"2024-04-09T21:40:34.327953+0000","last_became_active":"2024-04-09T21:40:33.332739+0000","last_became_peered":"2024-04-09T21:40:33.332739+0000","last_unstale":"2024-04-09T21:40:34.327953+0000","last_undegraded":"2024-04-09T21:40:34.327953+0000","last_fullsized":"2024-04-09T21:40:34.327953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T03:40:22.320370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337617+0000","last_change":"2024-04-09T21:40:33.332890+0000","last_active":"2024-04-09T21:40:33.337617+0000","last_peered":"2024-04-09T21:40:33.337617+0000","last_clean":"2024-04-09T21:40:33.337617+0000","last_became_active":"2024-04-09T21:40:33.332617+0000","last_became_peered":"2024-04-09T21:40:33.332617+0000","last_unstale":"2024-04-09T21:40:33.337617+0000","last_undegraded":"2024-04-09T21:40:33.337617+0000","last_fullsized":"2024-04-09T21:40:33.337617+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:57:01.831999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:40:33.337676+0000","last_change":"2024-04-09T21:40:33.332883+0000","last_active":"2024-04-09T21:40:33.337676+0000","last_peered":"2024-04-09T21:40:33.337676+0000","last_clean":"2024-04-09T21:40:33.337676+0000","last_became_active":"2024-04-09T21:40:33.332620+0000","last_became_peered":"2024-04-09T21:40:33.332620+0000","last_unstale":"2024-04-09T21:40:33.337676+0000","last_undegraded":"2024-04-09T21:40:33.337676+0000","last_fullsized":"2024-04-09T21:40:33.337676+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:32.318455+0000","last_clean_scrub_stamp":"2024-04-09T21:40:32.318455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T23:30:13.985736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27124,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830476,"statfs":{"total":107374182400,"available":107346407424,"internally_reserved":0,"allocated":98304,"data_stored":25270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28196,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829404,"statfs":{"total":107374182400,"available":107345309696,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27948,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829652,"statfs":{"total":107374182400,"available":107345563648,"internally_reserved":0,"allocated":692224,"data_stored":615638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-09T21:40:37.283 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-09T21:40:37.283 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-04-09T21:40:37.283 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-04-09T21:40:37.283 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-04-09T21:40:37.674 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:37.674 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-09T21:40:37.689 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-04-09T21:40:37.689 INFO:teuthology.run_tasks:Running task ceph... 2024-04-09T21:40:37.731 INFO:tasks.ceph:Creating ceph cluster c2... 2024-04-09T21:40:37.731 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-09T21:40:37.731 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7647917', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6358, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'targets': {'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6JmiQup3jBbUc4fDfxA1Cc87n92MyW1E37W5w047Bu63ZvbHk+9o3MyQjRiYMiRUTXFQavy0sy2i0bbdnktEc=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/q3UvERIx3e325Ay3goK8EiCuOC295xGHManCHbnvw67Gvt8s2Nmjbhmw1zIntD9LFijFSyuVpPD5AtVlfbmU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-09_01:21:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-09T21:40:37.732 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-09T21:40:37.734 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-09T21:40:37.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-04-09T21:40:37.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-04-09T21:40:37.841 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:37.841 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-04-09T21:40:37.859 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-09T21:40:37.859 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-04-09T21:40:37.918 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 179827 Links: 1 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-04-09 21:39:22.085916226 +0000 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-04-09 21:31:53.926480013 +0000 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-04-09 21:31:53.926480013 +0000 2024-04-09T21:40:37.919 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-04-09T21:40:37.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-09T21:40:37.987 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-04-09T21:40:37.987 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-04-09T21:40:37.987 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000264947 s, 1.9 MB/s 2024-04-09T21:40:37.988 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-09T21:40:38.046 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 176978 Links: 1 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-04-09 21:39:22.085916226 +0000 2024-04-09T21:40:38.104 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-04-09 21:31:54.282473385 +0000 2024-04-09T21:40:38.105 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-04-09 21:31:54.282473385 +0000 2024-04-09T21:40:38.105 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-04-09T21:40:38.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-09T21:40:38.170 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-04-09T21:40:38.171 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-04-09T21:40:38.171 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000242363 s, 2.1 MB/s 2024-04-09T21:40:38.172 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-09T21:40:38.230 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-04-09T21:40:38.287 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 182548 Links: 1 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-04-09 21:39:22.085916226 +0000 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-04-09 21:31:54.615467185 +0000 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-04-09 21:31:54.615467185 +0000 2024-04-09T21:40:38.288 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-04-09T21:40:38.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-09T21:40:38.353 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-04-09T21:40:38.353 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-04-09T21:40:38.353 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.00016864 s, 3.0 MB/s 2024-04-09T21:40:38.354 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-09T21:40:38.412 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 181673 Links: 1 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-04-09 21:39:22.085916226 +0000 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-04-09 21:31:54.945461042 +0000 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-04-09 21:31:54.945461042 +0000 2024-04-09T21:40:38.470 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-04-09T21:40:38.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-09T21:40:38.535 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-04-09T21:40:38.536 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-04-09T21:40:38.536 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000285591 s, 1.8 MB/s 2024-04-09T21:40:38.537 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-09T21:40:38.595 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:38.595 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-09T21:40:38.596 INFO:tasks.ceph:Generating config... 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-09T21:40:38.597 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-09T21:40:38.598 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-09T21:40:38.598 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-09T21:40:38.598 INFO:tasks.ceph:[client] setuser = ceph 2024-04-09T21:40:38.598 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-09T21:40:38.598 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-09T21:40:38.598 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-09T21:40:38.598 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-09T21:40:38.598 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-09T21:40:38.598 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-09T21:40:38.598 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-09T21:40:38.599 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-09T21:40:38.599 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-09T21:40:38.599 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-09T21:40:38.599 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-09T21:40:38.599 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-09T21:40:38.599 INFO:tasks.ceph:Setting up c2.mon.a... 2024-04-09T21:40:38.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-04-09T21:40:38.687 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c2.keyring 2024-04-09T21:40:38.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-04-09T21:40:38.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-09T21:40:38.762 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.137')] 2024-04-09T21:40:38.762 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.137'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-09T21:40:38.763 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:38.763 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-09T21:40:38.819 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.137 --print /home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:38.900 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:40:38.900 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: generated fsid 0aed30bd-1c79-4145-8ce5-301b0d17e710 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = pacific 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 0 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:fsid 0aed30bd-1c79-4145-8ce5-301b0d17e710 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:last_changed 2024-04-09T21:40:38.899537+0000 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-04-09T21:40:38.899537+0000 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:min_mon_release 16 (pacific) 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:election_strategy: 1 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.a 2024-04-09T21:40:38.901 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-04-09T21:40:38.902 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-09T21:40:38.959 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 0aed30bd-1c79-4145-8ce5-301b0d17e710... 2024-04-09T21:40:38.960 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-09T21:40:38.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-04-09T21:40:39.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-04-09T21:40:39.018 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-04-09T21:40:39.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = 172.21.15.137 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 0aed30bd-1c79-4145-8ce5-301b0d17e710 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-04-09T21:40:39.020 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-04-09T21:40:39.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-04-09T21:40:39.022 INFO:teuthology.orchestra.run.smithi063.stdout: rocksdb delete range threshold = 0 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-04-09T21:40:39.023 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-09T21:40:39.024 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-04-09T21:40:39.025 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw = 20 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw sync = 20 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt require ssl = False 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms backend = testing 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw curl low speed time = 300 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw data log num shards = 4 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw md log max shards = 4 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync data inject err probability = 0.1 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync log trim interval = 0 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync meta inject err probability = 0.1 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync obj etag verify = True 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: setgroup = ceph 2024-04-09T21:40:39.026 INFO:teuthology.orchestra.run.smithi063.stdout: setuser = ceph 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout: rgw data notify interval msec = 0 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_omap_entries_per_request = 10 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_omap_entries_per_request = 1000 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-04-09T21:40:39.027 INFO:teuthology.orchestra.run.smithi063.stdout: osd_max_omap_entries_per_request = 10000 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = 172.21.15.137 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 0aed30bd-1c79-4145-8ce5-301b0d17e710 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-04-09T21:40:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-04-09T21:40:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: rocksdb delete range threshold = 0 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-04-09T21:40:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-04-09T21:40:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:39.052 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw = 20 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: debug rgw sync = 20 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt require ssl = False 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms backend = testing 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw curl low speed time = 300 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data log num shards = 4 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw md log max shards = 4 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync data inject err probability = 0.1 2024-04-09T21:40:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync log trim interval = 0 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync meta inject err probability = 0.1 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: rgw sync obj etag verify = True 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: setgroup = ceph 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: setuser = ceph 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: rgw data notify interval msec = 0 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 1000 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-04-09T21:40:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: osd_max_omap_entries_per_request = 10000 2024-04-09T21:40:39.056 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-04-09T21:40:39.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-04-09T21:40:39.154 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-09T21:40:39.154 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:39.154 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-04-09T21:40:39.171 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:39.171 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-04-09T21:40:39.228 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-04-09T21:40:39.228 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:39.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/c2.keyring 2024-04-09T21:40:39.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-09T21:40:39.274 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-09T21:40:39.274 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:39.333 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-04-09T21:40:39.333 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:39.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/c2.keyring 2024-04-09T21:40:39.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-09T21:40:39.370 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:39.371 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:39.427 INFO:tasks.ceph:Setting up mon nodes... 2024-04-09T21:40:39.427 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-09T21:40:39.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-04-09T21:40:39.522 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-04-09T21:40:39.525 INFO:tasks.ceph:Setting up mds nodes... 2024-04-09T21:40:39.525 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-09T21:40:39.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-04-09T21:40:39.568 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c2.client.0.keyring 2024-04-09T21:40:39.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-04-09T21:40:39.625 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/c2.client.1.keyring 2024-04-09T21:40:39.638 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-09T21:40:39.638 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-09T21:40:39.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-04-09T21:40:39.706 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:39.706 INFO:tasks.ceph:role: c2.osd.0 2024-04-09T21:40:39.706 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-04-09T21:40:39.706 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-09T21:40:39.780 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-09T21:40:39.780 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-09T21:40:39.781 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-09T21:40:40.211 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-04-09T21:40:40.213 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-04-09T21:40:40.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-04-09T21:40:40.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-04-09T21:40:40.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-04-09T21:40:40.483 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:40.483 INFO:tasks.ceph:role: c2.osd.1 2024-04-09T21:40:40.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-04-09T21:40:40.484 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-09T21:40:40.557 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-09T21:40:40.558 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-09T21:40:40.982 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-04-09T21:40:40.984 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-04-09T21:40:40.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-04-09T21:40:41.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-04-09T21:40:41.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-04-09T21:40:41.185 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-09T21:40:41.185 INFO:tasks.ceph:role: c2.osd.2 2024-04-09T21:40:41.185 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-04-09T21:40:41.185 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-09T21:40:41.259 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-09T21:40:41.259 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-09T21:40:41.260 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-09T21:40:41.708 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-04-09T21:40:41.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-04-09T21:40:41.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-04-09T21:40:41.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-04-09T21:40:41.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:41.986 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:41.984+0000 7f1d4c46a7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-04-09T21:40:41.988 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:41.986+0000 7f1d4c46a7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-04-09T21:40:41.988 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:41.986+0000 7f1d4c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-09T21:40:41.989 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:41.986+0000 7f1d4c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-09T21:40:41.989 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:41.986+0000 7f1d4c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-09T21:40:41.996 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:41.993+0000 7f1d4c46a7c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-04-09T21:40:44.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-04-09T21:40:44.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:44.210 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:44.207+0000 7fa3d9d0d7c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-04-09T21:40:44.210 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:44.207+0000 7fa3d9d0d7c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-04-09T21:40:44.210 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:44.207+0000 7fa3d9d0d7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-09T21:40:44.210 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:44.207+0000 7fa3d9d0d7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-09T21:40:44.211 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:44.207+0000 7fa3d9d0d7c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-09T21:40:44.211 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:44.208+0000 7fa3d9d0d7c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-04-09T21:40:46.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-04-09T21:40:46.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:46.418 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:46.415+0000 7f27789797c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-04-09T21:40:46.418 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:46.416+0000 7f27789797c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-04-09T21:40:46.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:46.416+0000 7f27789797c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-09T21:40:46.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:46.416+0000 7f27789797c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-09T21:40:46.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:46.416+0000 7f27789797c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-09T21:40:46.419 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:40:46.416+0000 7f27789797c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-04-09T21:40:48.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-04-09T21:40:48.535 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-09T21:40:48.536 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:48.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-04-09T21:40:48.606 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:48.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-04-09T21:40:48.676 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:48.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-04-09T21:40:48.745 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:48.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-04-09T21:40:48.815 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:48.815 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-04-09T21:40:48.873 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:48.873 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-04-09T21:40:48.930 INFO:tasks.ceph:Adding keys to all mons... 2024-04-09T21:40:48.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/c2.keyring 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDXtRVmOqERHxAA8Q1m1Sg+3UbR5sCUa1hrZw== 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDZtRVmbma8OhAAafPmv/r/Yfq8QBOGM0ViMQ== 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDctRVmcq5tDBAAIGOs2zPIPkg3ORy//0o1+Q== 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDetRVmQdbaGBAAOumPt1usXBjPbBCp0yODPw== 2024-04-09T21:40:48.996 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-04-09T21:40:48.997 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDXtRVmXajTIRAAOKYPl5RsV9WEEFRD0DzpmQ== 2024-04-09T21:40:48.997 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-04-09T21:40:48.997 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDXtRVmvYMuJRAAwoC81ZjCh8QMVNdM0YuqBQ== 2024-04-09T21:40:48.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-09T21:40:49.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-09T21:40:49.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-09T21:40:49.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-09T21:40:49.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-09T21:40:49.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-09T21:40:49.367 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-09T21:40:49.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-04-09T21:40:49.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-04-09T21:40:49.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-04-09T21:40:49.523 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-04-09T21:40:49.580 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-04-09T21:40:49.580 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-04-09T21:40:49.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-04-09T21:40:49.622 INFO:tasks.ceph.c2.mon.a:Started 2024-04-09T21:40:49.623 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-04-09T21:40:49.623 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-04-09T21:40:49.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-04-09T21:40:49.625 INFO:tasks.ceph.c2.mgr.x:Started 2024-04-09T21:40:49.625 INFO:tasks.ceph:Setting crush tunables to default 2024-04-09T21:40:49.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd crush tunables default 2024-04-09T21:40:49.848 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-04-09T21:40:49.860 INFO:tasks.ceph:check_enable_crimson: False 2024-04-09T21:40:49.860 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-04-09T21:40:49.860 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:49.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-04-09T21:40:49.887 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:49.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-04-09T21:40:49.956 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-04-09T21:40:49.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-04-09T21:40:50.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd new 9912074b-2f50-439a-bbe1-0d2da10a0ad0 0 2024-04-09T21:40:50.282 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-04-09T21:40:50.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd new 88d6332b-e77b-4057-9f50-d9d0dae44009 1 2024-04-09T21:40:50.343 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:50.341+0000 7f807fbeb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-09T21:40:50.435 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:50.432+0000 7f807fbeb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-09T21:40:50.511 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:50.508+0000 7f807fbeb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-09T21:40:50.514 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-04-09T21:40:50.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd new d430e89c-f787-44fe-b903-c37b28632e03 2 2024-04-09T21:40:50.587 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:50.584+0000 7f807fbeb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-09T21:40:50.744 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-04-09T21:40:50.758 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-04-09T21:40:50.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-04-09T21:40:50.760 INFO:tasks.ceph.c2.osd.0:Started 2024-04-09T21:40:50.760 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-04-09T21:40:50.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-04-09T21:40:50.762 INFO:tasks.ceph.c2.osd.1:Started 2024-04-09T21:40:50.762 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-04-09T21:40:50.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-04-09T21:40:50.765 INFO:tasks.ceph.c2.osd.2:Started 2024-04-09T21:40:50.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-09T21:40:50.774 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:50.772+0000 7f807fbeb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-09T21:40:50.973 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:50.970+0000 7f807fbeb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-09T21:40:51.006 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:40:51.007 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":5,"fsid":"0aed30bd-1c79-4145-8ce5-301b0d17e710","created":"2024-04-09T21:40:49.749508+0000","modified":"2024-04-09T21:40:50.741309+0000","last_up_change":"0.000000","last_in_change":"2024-04-09T21:40:50.741309+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9912074b-2f50-439a-bbe1-0d2da10a0ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88d6332b-e77b-4057-9f50-d9d0dae44009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d430e89c-f787-44fe-b903-c37b28632e03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:51.022 INFO:tasks.ceph.ceph_manager.c2:[] 2024-04-09T21:40:51.022 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-09T21:40:51.088 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:51.085+0000 7f807fbeb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-09T21:40:51.184 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:51.181+0000 7f807fbeb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-09T21:40:51.260 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:51.257+0000 7f807fbeb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-09T21:40:51.324 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-09T21:40:51.370 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-04-09T21:40:51.367+0000 7f6f0d2ae7c0 -1 Falling back to public interface 2024-04-09T21:40:51.371 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-04-09T21:40:51.368+0000 7fcd9d2927c0 -1 Falling back to public interface 2024-04-09T21:40:51.382 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-04-09T21:40:51.379+0000 7f05085b77c0 -1 Falling back to public interface 2024-04-09T21:40:51.495 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:51.492+0000 7f807fbeb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-09T21:40:51.552 INFO:teuthology.misc.health.smithi137.stdout: 2024-04-09T21:40:51.552 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":5,"fsid":"0aed30bd-1c79-4145-8ce5-301b0d17e710","created":"2024-04-09T21:40:49.749508+0000","modified":"2024-04-09T21:40:50.741309+0000","last_up_change":"0.000000","last_in_change":"2024-04-09T21:40:50.741309+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9912074b-2f50-439a-bbe1-0d2da10a0ad0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"88d6332b-e77b-4057-9f50-d9d0dae44009","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d430e89c-f787-44fe-b903-c37b28632e03","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:51.564 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-09T21:40:51.569 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:51.566+0000 7f807fbeb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-09T21:40:51.958 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:51.956+0000 7f807fbeb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-09T21:40:52.281 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:52.279+0000 7f807fbeb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-09T21:40:52.441 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:52.438+0000 7f807fbeb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-09T21:40:52.521 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:52.518+0000 7f807fbeb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-09T21:40:53.220 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:53.217+0000 7f807fbeb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-09T21:40:53.304 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:53.302+0000 7f807fbeb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-09T21:40:53.435 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:53.433+0000 7f807fbeb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-09T21:40:53.998 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:53.996+0000 7f807fbeb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-09T21:40:54.070 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-04-09T21:40:54.067+0000 7fcd9d2927c0 -1 osd.0 0 log_to_monitors true 2024-04-09T21:40:54.093 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-04-09T21:40:54.090+0000 7f05085b77c0 -1 osd.2 0 log_to_monitors true 2024-04-09T21:40:54.093 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-04-09T21:40:54.090+0000 7f6f0d2ae7c0 -1 osd.1 0 log_to_monitors true 2024-04-09T21:40:54.180 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:54.178+0000 7f807fbeb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-09T21:40:54.344 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:54.341+0000 7f807fbeb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-09T21:40:55.158 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:55.156+0000 7f807fbeb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-09T21:40:55.430 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:55.428+0000 7f807fbeb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-09T21:40:55.528 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:55.525+0000 7f807fbeb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-09T21:40:55.949 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:55.947+0000 7f807fbeb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-09T21:40:56.333 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-04-09T21:40:56.331+0000 7fcd9164a700 -1 osd.0 0 waiting for initial osdmap 2024-04-09T21:40:56.344 INFO:tasks.ceph.c2.osd.0.smithi137.stderr:2024-04-09T21:40:56.342+0000 7fcd89224700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-09T21:40:56.347 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-04-09T21:40:56.344+0000 7f6f01666700 -1 osd.1 0 waiting for initial osdmap 2024-04-09T21:40:56.354 INFO:tasks.ceph.c2.osd.1.smithi137.stderr:2024-04-09T21:40:56.352+0000 7f6ef9240700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-09T21:40:56.357 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-04-09T21:40:56.354+0000 7f04fc96f700 -1 osd.2 0 waiting for initial osdmap 2024-04-09T21:40:56.371 INFO:tasks.ceph.c2.osd.2.smithi137.stderr:2024-04-09T21:40:56.369+0000 7f04f4549700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-09T21:40:57.008 INFO:tasks.ceph.c2.mgr.x.smithi137.stderr:2024-04-09T21:40:57.005+0000 7f803ae65700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-09T21:40:57.867 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-09T21:40:58.198 INFO:teuthology.misc.health.smithi137.stdout: 2024-04-09T21:40:58.198 INFO:teuthology.misc.health.smithi137.stdout:{"epoch":8,"fsid":"0aed30bd-1c79-4145-8ce5-301b0d17e710","created":"2024-04-09T21:40:49.749508+0000","modified":"2024-04-09T21:40:57.346417+0000","last_up_change":"2024-04-09T21:40:57.346417+0000","last_in_change":"2024-04-09T21:40:50.741309+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9912074b-2f50-439a-bbe1-0d2da10a0ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":40907},{"type":"v1","addr":"172.21.15.137:6809","nonce":40907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":40907},{"type":"v1","addr":"172.21.15.137:6811","nonce":40907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":40907},{"type":"v1","addr":"172.21.15.137:6815","nonce":40907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":40907},{"type":"v1","addr":"172.21.15.137:6813","nonce":40907}]},"public_addr":"172.21.15.137:6809/40907","cluster_addr":"172.21.15.137:6811/40907","heartbeat_back_addr":"172.21.15.137:6815/40907","heartbeat_front_addr":"172.21.15.137:6813/40907","state":["exists","up"]},{"osd":1,"uuid":"88d6332b-e77b-4057-9f50-d9d0dae44009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":40908},{"type":"v1","addr":"172.21.15.137:6801","nonce":40908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":40908},{"type":"v1","addr":"172.21.15.137:6803","nonce":40908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":40908},{"type":"v1","addr":"172.21.15.137:6807","nonce":40908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":40908},{"type":"v1","addr":"172.21.15.137:6805","nonce":40908}]},"public_addr":"172.21.15.137:6801/40908","cluster_addr":"172.21.15.137:6803/40908","heartbeat_back_addr":"172.21.15.137:6807/40908","heartbeat_front_addr":"172.21.15.137:6805/40908","state":["exists","up"]},{"osd":2,"uuid":"d430e89c-f787-44fe-b903-c37b28632e03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":40920},{"type":"v1","addr":"172.21.15.137:6817","nonce":40920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":40920},{"type":"v1","addr":"172.21.15.137:6819","nonce":40920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":40920},{"type":"v1","addr":"172.21.15.137:6823","nonce":40920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":40920},{"type":"v1","addr":"172.21.15.137:6821","nonce":40920}]},"public_addr":"172.21.15.137:6817/40920","cluster_addr":"172.21.15.137:6819/40920","heartbeat_back_addr":"172.21.15.137:6823/40920","heartbeat_front_addr":"172.21.15.137:6821/40920","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:40:58.212 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-09T21:40:58.213 INFO:tasks.ceph:Creating RBD pool 2024-04-09T21:40:58.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd pool create rbd 8 2024-04-09T21:40:59.014 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'rbd' created 2024-04-09T21:40:59.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-09T21:41:00.019 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-09T21:41:00.035 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-04-09T21:41:00.036 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-04-09T21:41:00.036 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-04-09T21:41:00.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-04-09T21:41:00.037 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-09T21:41:00.466 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:41:00.480 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":40528},{"type":"v1","addr":"172.21.15.137:6825","nonce":40528}]},"active_addr":"172.21.15.137:6825/40528","active_change":"2024-04-09T21:40:55.960946+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1747375543}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3773588448}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1979055255}]}]} 2024-04-09T21:41:00.484 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-04-09T21:41:00.484 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-04-09T21:41:00.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-09T21:41:00.813 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:41:00.814 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":10,"fsid":"0aed30bd-1c79-4145-8ce5-301b0d17e710","created":"2024-04-09T21:40:49.749508+0000","modified":"2024-04-09T21:41:00.005880+0000","last_up_change":"2024-04-09T21:40:57.346417+0000","last_in_change":"2024-04-09T21:40:50.741309+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-09T21:40:58.041268+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-09T21:40:58.553277+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9912074b-2f50-439a-bbe1-0d2da10a0ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":40907},{"type":"v1","addr":"172.21.15.137:6809","nonce":40907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":40907},{"type":"v1","addr":"172.21.15.137:6811","nonce":40907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":40907},{"type":"v1","addr":"172.21.15.137:6815","nonce":40907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":40907},{"type":"v1","addr":"172.21.15.137:6813","nonce":40907}]},"public_addr":"172.21.15.137:6809/40907","cluster_addr":"172.21.15.137:6811/40907","heartbeat_back_addr":"172.21.15.137:6815/40907","heartbeat_front_addr":"172.21.15.137:6813/40907","state":["exists","up"]},{"osd":1,"uuid":"88d6332b-e77b-4057-9f50-d9d0dae44009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":40908},{"type":"v1","addr":"172.21.15.137:6801","nonce":40908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":40908},{"type":"v1","addr":"172.21.15.137:6803","nonce":40908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":40908},{"type":"v1","addr":"172.21.15.137:6807","nonce":40908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":40908},{"type":"v1","addr":"172.21.15.137:6805","nonce":40908}]},"public_addr":"172.21.15.137:6801/40908","cluster_addr":"172.21.15.137:6803/40908","heartbeat_back_addr":"172.21.15.137:6807/40908","heartbeat_front_addr":"172.21.15.137:6805/40908","state":["exists","up"]},{"osd":2,"uuid":"d430e89c-f787-44fe-b903-c37b28632e03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":40920},{"type":"v1","addr":"172.21.15.137:6817","nonce":40920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":40920},{"type":"v1","addr":"172.21.15.137:6819","nonce":40920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":40920},{"type":"v1","addr":"172.21.15.137:6823","nonce":40920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":40920},{"type":"v1","addr":"172.21.15.137:6821","nonce":40920}]},"public_addr":"172.21.15.137:6817/40920","cluster_addr":"172.21.15.137:6819/40920","heartbeat_back_addr":"172.21.15.137:6823/40920","heartbeat_front_addr":"172.21.15.137:6821/40920","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:55.091275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:55.098989+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:55.112681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:41:00.827 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-04-09T21:41:00.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-09T21:41:01.152 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:41:01.152 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":11,"fsid":"0aed30bd-1c79-4145-8ce5-301b0d17e710","created":"2024-04-09T21:40:49.749508+0000","modified":"2024-04-09T21:41:01.006657+0000","last_up_change":"2024-04-09T21:40:57.346417+0000","last_in_change":"2024-04-09T21:40:50.741309+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-09T21:40:58.041268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-09T21:40:58.553277+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9912074b-2f50-439a-bbe1-0d2da10a0ad0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":40907},{"type":"v1","addr":"172.21.15.137:6809","nonce":40907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":40907},{"type":"v1","addr":"172.21.15.137:6811","nonce":40907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":40907},{"type":"v1","addr":"172.21.15.137:6815","nonce":40907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":40907},{"type":"v1","addr":"172.21.15.137:6813","nonce":40907}]},"public_addr":"172.21.15.137:6809/40907","cluster_addr":"172.21.15.137:6811/40907","heartbeat_back_addr":"172.21.15.137:6815/40907","heartbeat_front_addr":"172.21.15.137:6813/40907","state":["exists","up"]},{"osd":1,"uuid":"88d6332b-e77b-4057-9f50-d9d0dae44009","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":40908},{"type":"v1","addr":"172.21.15.137:6801","nonce":40908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":40908},{"type":"v1","addr":"172.21.15.137:6803","nonce":40908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":40908},{"type":"v1","addr":"172.21.15.137:6807","nonce":40908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":40908},{"type":"v1","addr":"172.21.15.137:6805","nonce":40908}]},"public_addr":"172.21.15.137:6801/40908","cluster_addr":"172.21.15.137:6803/40908","heartbeat_back_addr":"172.21.15.137:6807/40908","heartbeat_front_addr":"172.21.15.137:6805/40908","state":["exists","up"]},{"osd":2,"uuid":"d430e89c-f787-44fe-b903-c37b28632e03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":40920},{"type":"v1","addr":"172.21.15.137:6817","nonce":40920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":40920},{"type":"v1","addr":"172.21.15.137:6819","nonce":40920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":40920},{"type":"v1","addr":"172.21.15.137:6823","nonce":40920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":40920},{"type":"v1","addr":"172.21.15.137:6821","nonce":40920}]},"public_addr":"172.21.15.137:6817/40920","cluster_addr":"172.21.15.137:6819/40920","heartbeat_back_addr":"172.21.15.137:6823/40920","heartbeat_front_addr":"172.21.15.137:6821/40920","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:55.091275+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:55.098989+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-09T21:40:55.112681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-09T21:41:01.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-04-09T21:41:01.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-04-09T21:41:01.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-04-09T21:41:01.333 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-04-09T21:41:01.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-09T21:41:01.359 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-04-09T21:41:01.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-09T21:41:01.364 INFO:teuthology.orchestra.run.smithi137.stdout:34359738371 2024-04-09T21:41:01.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-09T21:41:01.684 INFO:teuthology.orchestra.run.smithi137.stdout:34359738369 2024-04-09T21:41:01.698 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2024-04-09T21:41:01.750 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-04-09T21:41:01.751 INFO:teuthology.orchestra.run.smithi137.stdout:34359738369 2024-04-09T21:41:01.766 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.2 2024-04-09T21:41:01.767 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.0 2024-04-09T21:41:02.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-09T21:41:02.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-09T21:41:02.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-09T21:41:03.060 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-04-09T21:41:03.075 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2024-04-09T21:41:03.075 DEBUG:teuthology.parallel:result is None 2024-04-09T21:41:03.137 INFO:teuthology.orchestra.run.smithi137.stdout:34359738371 2024-04-09T21:41:03.143 INFO:teuthology.orchestra.run.smithi137.stdout:34359738370 2024-04-09T21:41:03.152 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2024-04-09T21:41:03.152 DEBUG:teuthology.parallel:result is None 2024-04-09T21:41:03.158 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.0 2024-04-09T21:41:03.158 DEBUG:teuthology.parallel:result is None 2024-04-09T21:41:03.158 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-09T21:41:03.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-09T21:41:03.522 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:41:03.524 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-04-09T21:41:03.535 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-09T21:41:02.003365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82616,"kb_used_data":1428,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490184,"statfs":{"total":322122547200,"available":322037948416,"internally_reserved":0,"allocated":1462272,"data_stored":1251213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4779,"internal_metadata":82963797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008957+0000","last_change":"2024-04-09T21:41:00.015963+0000","last_active":"2024-04-09T21:41:01.008957+0000","last_peered":"2024-04-09T21:41:01.008957+0000","last_clean":"2024-04-09T21:41:01.008957+0000","last_became_active":"2024-04-09T21:41:00.015684+0000","last_became_peered":"2024-04-09T21:41:00.015684+0000","last_unstale":"2024-04-09T21:41:01.008957+0000","last_undegraded":"2024-04-09T21:41:01.008957+0000","last_fullsized":"2024-04-09T21:41:01.008957+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T23:57:34.196131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008871+0000","last_change":"2024-04-09T21:41:00.016636+0000","last_active":"2024-04-09T21:41:01.008871+0000","last_peered":"2024-04-09T21:41:01.008871+0000","last_clean":"2024-04-09T21:41:01.008871+0000","last_became_active":"2024-04-09T21:41:00.016381+0000","last_became_peered":"2024-04-09T21:41:00.016381+0000","last_unstale":"2024-04-09T21:41:01.008871+0000","last_undegraded":"2024-04-09T21:41:01.008871+0000","last_fullsized":"2024-04-09T21:41:01.008871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:28:25.830523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008886+0000","last_change":"2024-04-09T21:41:00.015231+0000","last_active":"2024-04-09T21:41:01.008886+0000","last_peered":"2024-04-09T21:41:01.008886+0000","last_clean":"2024-04-09T21:41:01.008886+0000","last_became_active":"2024-04-09T21:41:00.014973+0000","last_became_peered":"2024-04-09T21:41:00.014973+0000","last_unstale":"2024-04-09T21:41:01.008886+0000","last_undegraded":"2024-04-09T21:41:01.008886+0000","last_fullsized":"2024-04-09T21:41:01.008886+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T05:57:47.528618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-09T21:41:00.014470+0000","last_change":"2024-04-09T21:41:00.013426+0000","last_active":"2024-04-09T21:41:00.014470+0000","last_peered":"2024-04-09T21:41:00.014470+0000","last_clean":"2024-04-09T21:41:00.014470+0000","last_became_active":"2024-04-09T21:41:00.013131+0000","last_became_peered":"2024-04-09T21:41:00.013131+0000","last_unstale":"2024-04-09T21:41:00.014470+0000","last_undegraded":"2024-04-09T21:41:00.014470+0000","last_fullsized":"2024-04-09T21:41:00.014470+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T06:10:17.398339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008805+0000","last_change":"2024-04-09T21:41:00.017319+0000","last_active":"2024-04-09T21:41:01.008805+0000","last_peered":"2024-04-09T21:41:01.008805+0000","last_clean":"2024-04-09T21:41:01.008805+0000","last_became_active":"2024-04-09T21:41:00.016416+0000","last_became_peered":"2024-04-09T21:41:00.016416+0000","last_unstale":"2024-04-09T21:41:01.008805+0000","last_undegraded":"2024-04-09T21:41:01.008805+0000","last_fullsized":"2024-04-09T21:41:01.008805+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T00:20:19.090740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.009022+0000","last_change":"2024-04-09T21:41:00.013245+0000","last_active":"2024-04-09T21:41:01.009022+0000","last_peered":"2024-04-09T21:41:01.009022+0000","last_clean":"2024-04-09T21:41:01.009022+0000","last_became_active":"2024-04-09T21:41:00.012994+0000","last_became_peered":"2024-04-09T21:41:00.012994+0000","last_unstale":"2024-04-09T21:41:01.009022+0000","last_undegraded":"2024-04-09T21:41:01.009022+0000","last_fullsized":"2024-04-09T21:41:01.009022+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T05:53:56.602089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-09T21:41:00.014463+0000","last_change":"2024-04-09T21:41:00.013417+0000","last_active":"2024-04-09T21:41:00.014463+0000","last_peered":"2024-04-09T21:41:00.014463+0000","last_clean":"2024-04-09T21:41:00.014463+0000","last_became_active":"2024-04-09T21:41:00.013141+0000","last_became_peered":"2024-04-09T21:41:00.013141+0000","last_unstale":"2024-04-09T21:41:00.014463+0000","last_undegraded":"2024-04-09T21:41:00.014463+0000","last_fullsized":"2024-04-09T21:41:00.014463+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T05:39:31.130380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-09T21:41:00.015691+0000","last_change":"2024-04-09T21:41:00.014867+0000","last_active":"2024-04-09T21:41:00.015691+0000","last_peered":"2024-04-09T21:41:00.015691+0000","last_clean":"2024-04-09T21:41:00.015691+0000","last_became_active":"2024-04-09T21:41:00.014587+0000","last_became_peered":"2024-04-09T21:41:00.014587+0000","last_unstale":"2024-04-09T21:41:00.015691+0000","last_undegraded":"2024-04-09T21:41:00.015691+0000","last_fullsized":"2024-04-09T21:41:00.015691+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:21:59.872769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008928+0000","last_change":"2024-04-09T21:41:00.015845+0000","last_active":"2024-04-09T21:41:01.008928+0000","last_peered":"2024-04-09T21:41:01.008928+0000","last_clean":"2024-04-09T21:41:01.008928+0000","last_became_active":"2024-04-09T21:41:00.015582+0000","last_became_peered":"2024-04-09T21:41:00.015582+0000","last_unstale":"2024-04-09T21:41:01.008928+0000","last_undegraded":"2024-04-09T21:41:01.008928+0000","last_fullsized":"2024-04-09T21:41:01.008928+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T04:35:59.404302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830488,"statfs":{"total":107374182400,"available":107346419712,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27808,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829792,"statfs":{"total":107374182400,"available":107345707008,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-09T21:41:03.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-09T21:41:03.854 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:41:03.856 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-04-09T21:41:03.868 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-09T21:41:02.003365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82616,"kb_used_data":1428,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314490184,"statfs":{"total":322122547200,"available":322037948416,"internally_reserved":0,"allocated":1462272,"data_stored":1251213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4779,"internal_metadata":82963797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008957+0000","last_change":"2024-04-09T21:41:00.015963+0000","last_active":"2024-04-09T21:41:01.008957+0000","last_peered":"2024-04-09T21:41:01.008957+0000","last_clean":"2024-04-09T21:41:01.008957+0000","last_became_active":"2024-04-09T21:41:00.015684+0000","last_became_peered":"2024-04-09T21:41:00.015684+0000","last_unstale":"2024-04-09T21:41:01.008957+0000","last_undegraded":"2024-04-09T21:41:01.008957+0000","last_fullsized":"2024-04-09T21:41:01.008957+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-10T23:57:34.196131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008871+0000","last_change":"2024-04-09T21:41:00.016636+0000","last_active":"2024-04-09T21:41:01.008871+0000","last_peered":"2024-04-09T21:41:01.008871+0000","last_clean":"2024-04-09T21:41:01.008871+0000","last_became_active":"2024-04-09T21:41:00.016381+0000","last_became_peered":"2024-04-09T21:41:00.016381+0000","last_unstale":"2024-04-09T21:41:01.008871+0000","last_undegraded":"2024-04-09T21:41:01.008871+0000","last_fullsized":"2024-04-09T21:41:01.008871+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:28:25.830523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008886+0000","last_change":"2024-04-09T21:41:00.015231+0000","last_active":"2024-04-09T21:41:01.008886+0000","last_peered":"2024-04-09T21:41:01.008886+0000","last_clean":"2024-04-09T21:41:01.008886+0000","last_became_active":"2024-04-09T21:41:00.014973+0000","last_became_peered":"2024-04-09T21:41:00.014973+0000","last_unstale":"2024-04-09T21:41:01.008886+0000","last_undegraded":"2024-04-09T21:41:01.008886+0000","last_fullsized":"2024-04-09T21:41:01.008886+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T05:57:47.528618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-09T21:41:00.014470+0000","last_change":"2024-04-09T21:41:00.013426+0000","last_active":"2024-04-09T21:41:00.014470+0000","last_peered":"2024-04-09T21:41:00.014470+0000","last_clean":"2024-04-09T21:41:00.014470+0000","last_became_active":"2024-04-09T21:41:00.013131+0000","last_became_peered":"2024-04-09T21:41:00.013131+0000","last_unstale":"2024-04-09T21:41:00.014470+0000","last_undegraded":"2024-04-09T21:41:00.014470+0000","last_fullsized":"2024-04-09T21:41:00.014470+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T06:10:17.398339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008805+0000","last_change":"2024-04-09T21:41:00.017319+0000","last_active":"2024-04-09T21:41:01.008805+0000","last_peered":"2024-04-09T21:41:01.008805+0000","last_clean":"2024-04-09T21:41:01.008805+0000","last_became_active":"2024-04-09T21:41:00.016416+0000","last_became_peered":"2024-04-09T21:41:00.016416+0000","last_unstale":"2024-04-09T21:41:01.008805+0000","last_undegraded":"2024-04-09T21:41:01.008805+0000","last_fullsized":"2024-04-09T21:41:01.008805+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T00:20:19.090740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.009022+0000","last_change":"2024-04-09T21:41:00.013245+0000","last_active":"2024-04-09T21:41:01.009022+0000","last_peered":"2024-04-09T21:41:01.009022+0000","last_clean":"2024-04-09T21:41:01.009022+0000","last_became_active":"2024-04-09T21:41:00.012994+0000","last_became_peered":"2024-04-09T21:41:00.012994+0000","last_unstale":"2024-04-09T21:41:01.009022+0000","last_undegraded":"2024-04-09T21:41:01.009022+0000","last_fullsized":"2024-04-09T21:41:01.009022+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T05:53:56.602089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-09T21:41:00.014463+0000","last_change":"2024-04-09T21:41:00.013417+0000","last_active":"2024-04-09T21:41:00.014463+0000","last_peered":"2024-04-09T21:41:00.014463+0000","last_clean":"2024-04-09T21:41:00.014463+0000","last_became_active":"2024-04-09T21:41:00.013141+0000","last_became_peered":"2024-04-09T21:41:00.013141+0000","last_unstale":"2024-04-09T21:41:00.014463+0000","last_undegraded":"2024-04-09T21:41:00.014463+0000","last_fullsized":"2024-04-09T21:41:00.014463+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T05:39:31.130380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-04-09T21:41:00.015691+0000","last_change":"2024-04-09T21:41:00.014867+0000","last_active":"2024-04-09T21:41:00.015691+0000","last_peered":"2024-04-09T21:41:00.015691+0000","last_clean":"2024-04-09T21:41:00.015691+0000","last_became_active":"2024-04-09T21:41:00.014587+0000","last_became_peered":"2024-04-09T21:41:00.014587+0000","last_unstale":"2024-04-09T21:41:00.015691+0000","last_undegraded":"2024-04-09T21:41:00.015691+0000","last_fullsized":"2024-04-09T21:41:00.015691+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T02:21:59.872769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-09T21:41:01.008928+0000","last_change":"2024-04-09T21:41:00.015845+0000","last_active":"2024-04-09T21:41:01.008928+0000","last_peered":"2024-04-09T21:41:01.008928+0000","last_clean":"2024-04-09T21:41:01.008928+0000","last_became_active":"2024-04-09T21:41:00.015582+0000","last_became_peered":"2024-04-09T21:41:00.015582+0000","last_unstale":"2024-04-09T21:41:01.008928+0000","last_undegraded":"2024-04-09T21:41:01.008928+0000","last_fullsized":"2024-04-09T21:41:01.008928+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-09T21:40:59.004992+0000","last_clean_scrub_stamp":"2024-04-09T21:40:59.004992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-11T04:35:59.404302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27112,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830488,"statfs":{"total":107374182400,"available":107346419712,"internally_reserved":0,"allocated":90112,"data_stored":22580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27808,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829792,"statfs":{"total":107374182400,"available":107345707008,"internally_reserved":0,"allocated":679936,"data_stored":611837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829904,"statfs":{"total":107374182400,"available":107345821696,"internally_reserved":0,"allocated":692224,"data_stored":616796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-09T21:41:03.869 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-04-09T21:41:03.869 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-04-09T21:41:03.869 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-04-09T21:41:03.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-04-09T21:41:04.237 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-04-09T21:41:04.238 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-09T21:41:04.251 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-04-09T21:41:04.252 INFO:teuthology.run_tasks:Running task rgw... 2024-04-09T21:41:04.265 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-04-09T21:41:04.266 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-04-09T21:41:04.266 INFO:tasks.rgw:Creating data pools 2024-04-09T21:41:04.266 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-04-09T21:41:04.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-09T21:41:05.364 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.data' created 2024-04-09T21:41:05.382 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-09T21:41:06.378 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-09T21:41:06.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-09T21:41:07.379 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.index' created 2024-04-09T21:41:07.394 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-09T21:41:08.392 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-09T21:41:08.410 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-04-09T21:41:08.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-09T21:41:08.758 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-09T21:41:08.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-09T21:41:09.382 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-09T21:41:09.394 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-09T21:41:09.739 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-09T21:41:09.751 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-09T21:41:10.392 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-09T21:41:10.404 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-04-09T21:41:10.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-09T21:41:11.393 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.data' created 2024-04-09T21:41:11.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-09T21:41:12.398 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-09T21:41:12.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-09T21:41:13.400 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.index' created 2024-04-09T21:41:13.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-09T21:41:14.410 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-09T21:41:14.432 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-04-09T21:41:14.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-09T21:41:14.761 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-09T21:41:14.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-09T21:41:15.400 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-09T21:41:15.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-09T21:41:15.738 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-09T21:41:15.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-09T21:41:16.406 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-09T21:41:16.417 DEBUG:tasks.rgw:Pools created 2024-04-09T21:41:16.417 INFO:tasks.rgw:Configuring compression type = random 2024-04-09T21:41:16.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-04-09T21:41:16.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-04-09T21:41:16.418 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-04-09T21:41:16.496 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:16.496 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:16.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:16.504+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.405+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.405+0000 7f9415ed6780 20 realm 2024-04-09T21:41:18.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.405+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.406+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.406+0000 7f9415ed6780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-09T21:41:18.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.406+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.408+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.408+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.408+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.408+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.409+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.409+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.410+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.410+0000 7f9415ed6780 10 Creating default zonegroup 2024-04-09T21:41:18.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.410+0000 7f9415ed6780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-09T21:41:18.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.420+0000 7f9415ed6780 10 WARNING: driver->list_zones() returned r=-2 2024-04-09T21:41:18.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.420+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.420+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.424+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.424+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.424+0000 7f9415ed6780 10 could not read realm id: (2) No such file or directory 2024-04-09T21:41:18.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.424+0000 7f9415ed6780 10 WARNING: failed to set zone as default, r=-22 2024-04-09T21:41:18.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.424+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.424+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.426+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.427+0000 7f9415ed6780 20 rados_obj.operate() r=0 bl.length=874 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.427+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.427+0000 7f9415ed6780 20 rados_obj.operate() r=0 bl.length=420 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.427+0000 7f9415ed6780 10 Using default name default 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.427+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 rados_obj.operate() r=0 bl.length=46 2024-04-09T21:41:18.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 rados_obj.operate() r=0 bl.length=874 2024-04-09T21:41:18.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 zone default found 2024-04-09T21:41:18.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 4 Realm: () 2024-04-09T21:41:18.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 4 ZoneGroup: default (31117561-1868-4e8c-9568-16af99f5ac15) 2024-04-09T21:41:18.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 4 Zone: default (32720c28-1ce2-4569-b31f-12fb91c8f57a) 2024-04-09T21:41:18.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-09T21:41:18.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 zonegroup default 2024-04-09T21:41:18.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:18.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:18.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:18.428+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:20.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:20.434+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:20.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:20.434+0000 7f9415ed6780 20 rados->read ofs=0 len=0 2024-04-09T21:41:20.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:20.435+0000 7f9415ed6780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:20.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:20.435+0000 7f9415ed6780 20 started sync module instance, tier type = 2024-04-09T21:41:20.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:20.435+0000 7f9415ed6780 20 started zone id=32720c28-1ce2-4569-b31f-12fb91c8f57a (name=default) with tier type = 2024-04-09T21:41:22.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=0 2024-04-09T21:41:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=1 2024-04-09T21:41:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=2 2024-04-09T21:41:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=3 2024-04-09T21:41:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=4 2024-04-09T21:41:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=5 2024-04-09T21:41:22.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.462+0000 7f9415ed6780 20 add_watcher() i=6 2024-04-09T21:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.464+0000 7f9415ed6780 20 add_watcher() i=7 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.464+0000 7f9415ed6780 2 all 8 watchers are set, enabling cache 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-09T21:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-09T21:41:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-09T21:41:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.465+0000 7f9415ed6780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.466+0000 7f9415ed6780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-09T21:41:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.466+0000 7f9415ed6780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-09T21:41:22.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.467+0000 7f9415ed6780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:22.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.470+0000 7f93fae51700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:22.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.470+0000 7f9415ed6780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:22.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:22.470+0000 7f9415ed6780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:25.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.435+0000 7f9415ed6780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-09T21:41:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.435+0000 7f9415ed6780 5 note: GC not initialized 2024-04-09T21:41:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.437+0000 7f93f6648700 20 reqs_thread_entry: start 2024-04-09T21:41:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.437+0000 7f9415ed6780 20 init_complete bucket index max shards: 11 2024-04-09T21:41:25.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.437+0000 7f9415ed6780 10 rgw notify: Started notification manager with: 1 workers 2024-04-09T21:41:25.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.438+0000 7f93f4644700 20 rgw notify: INFO: next queues processing will happen at: Tue Apr 9 21:41:55 2024 2024-04-09T21:41:25.440 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-09T21:41:25.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.438+0000 7f93f3e43700 20 reqs_thread_entry: start 2024-04-09T21:41:25.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.462+0000 7f93fae51700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:25.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.464+0000 7f9415ed6780 20 remove_watcher() i=0 2024-04-09T21:41:25.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.464+0000 7f9415ed6780 2 removed watcher, disabling cache 2024-04-09T21:41:25.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.466+0000 7f9415ed6780 20 remove_watcher() i=1 2024-04-09T21:41:25.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.468+0000 7f9415ed6780 20 remove_watcher() i=2 2024-04-09T21:41:25.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.471+0000 7f9415ed6780 20 remove_watcher() i=3 2024-04-09T21:41:25.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.474+0000 7f9415ed6780 20 remove_watcher() i=4 2024-04-09T21:41:25.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.475+0000 7f9415ed6780 20 remove_watcher() i=5 2024-04-09T21:41:25.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.477+0000 7f9415ed6780 20 remove_watcher() i=6 2024-04-09T21:41:25.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.480+0000 7f9415ed6780 20 remove_watcher() i=7 2024-04-09T21:41:25.485 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2024-04-09T21:41:25.487 DEBUG:tasks.util.rgw: json result: [] 2024-04-09T21:41:25.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:25.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:25.487 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-09T21:41:25.541 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:25.541 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.549+0000 7fcec2e07780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.549+0000 7fcec2e07780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:25.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.554+0000 7fcea457b700 20 reqs_thread_entry: start 2024-04-09T21:41:25.562 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:25.562 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "32720c28-1ce2-4569-b31f-12fb91c8f57a", 2024-04-09T21:41:25.562 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default", 2024-04-09T21:41:25.562 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "default.rgw.meta:root", 2024-04-09T21:41:25.562 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "default.rgw.control", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "default.rgw.log", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "default.rgw.otp", 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T21:41:25.563 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "", 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "" 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "compression_type": "random" 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-09T21:41:25.564 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "", 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-09T21:41:25.565 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:25.566 DEBUG:tasks.util.rgw: json result: {'id': '32720c28-1ce2-4569-b31f-12fb91c8f57a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-09T21:41:25.566 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-04-09T21:41:25.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-04-09T21:41:25.567 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-04-09T21:41:25.611 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:25.611 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:25.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.619+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.619+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.619+0000 7fd48d8b4780 20 realm 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.619+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 20 rados_obj.operate() r=0 bl.length=46 2024-04-09T21:41:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.620+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.621+0000 7fd48d8b4780 20 rados_obj.operate() r=0 bl.length=890 2024-04-09T21:41:25.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.621+0000 7fd48d8b4780 20 searching for the correct realm 2024-04-09T21:41:25.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 RGWRados::pool_iterate: got zonegroup_info.31117561-1868-4e8c-9568-16af99f5ac15 2024-04-09T21:41:25.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 RGWRados::pool_iterate: got zone_info.32720c28-1ce2-4569-b31f-12fb91c8f57a 2024-04-09T21:41:25.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados_obj.operate() r=0 bl.length=46 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados_obj.operate() r=0 bl.length=420 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 zone default found 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 4 Realm: () 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 4 ZoneGroup: default (31117561-1868-4e8c-9568-16af99f5ac15) 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 4 Zone: default (32720c28-1ce2-4569-b31f-12fb91c8f57a) 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 zonegroup default 2024-04-09T21:41:25.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.628+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 rados->read ofs=0 len=0 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 started sync module instance, tier type = 2024-04-09T21:41:25.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.629+0000 7fd48d8b4780 20 started zone id=32720c28-1ce2-4569-b31f-12fb91c8f57a (name=default) with tier type = 2024-04-09T21:41:25.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=0 2024-04-09T21:41:25.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=1 2024-04-09T21:41:25.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=2 2024-04-09T21:41:25.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=3 2024-04-09T21:41:25.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=4 2024-04-09T21:41:25.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=5 2024-04-09T21:41:25.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.662+0000 7fd48d8b4780 20 add_watcher() i=6 2024-04-09T21:41:25.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.664+0000 7fd48d8b4780 20 add_watcher() i=7 2024-04-09T21:41:25.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.664+0000 7fd48d8b4780 2 all 8 watchers are set, enabling cache 2024-04-09T21:41:25.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.668+0000 7fd47282f700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:25.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.668+0000 7fd48d8b4780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:25.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.668+0000 7fd48d8b4780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:25.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.672+0000 7fd48d8b4780 5 note: GC not initialized 2024-04-09T21:41:25.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.672+0000 7fd46e026700 20 reqs_thread_entry: start 2024-04-09T21:41:25.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.673+0000 7fd48d8b4780 20 init_complete bucket index max shards: 11 2024-04-09T21:41:25.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.673+0000 7fd48d8b4780 10 rgw notify: Started notification manager with: 1 workers 2024-04-09T21:41:25.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.673+0000 7fd46c022700 20 rgw notify: INFO: next queues processing will happen at: Tue Apr 9 21:41:55 2024 2024-04-09T21:41:25.675 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-09T21:41:25.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.673+0000 7fd46b800700 20 reqs_thread_entry: start 2024-04-09T21:41:25.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.680+0000 7fd47282f700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:25.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.684+0000 7fd48d8b4780 20 remove_watcher() i=0 2024-04-09T21:41:25.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.684+0000 7fd48d8b4780 2 removed watcher, disabling cache 2024-04-09T21:41:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.687+0000 7fd48d8b4780 20 remove_watcher() i=1 2024-04-09T21:41:25.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.688+0000 7fd48d8b4780 20 remove_watcher() i=2 2024-04-09T21:41:25.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.690+0000 7fd48d8b4780 20 remove_watcher() i=3 2024-04-09T21:41:25.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.693+0000 7fd48d8b4780 20 remove_watcher() i=4 2024-04-09T21:41:25.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.694+0000 7fd48d8b4780 20 remove_watcher() i=5 2024-04-09T21:41:25.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.695+0000 7fd48d8b4780 20 remove_watcher() i=6 2024-04-09T21:41:25.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.698+0000 7fd48d8b4780 20 remove_watcher() i=7 2024-04-09T21:41:25.702 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2024-04-09T21:41:25.704 DEBUG:tasks.util.rgw: json result: [] 2024-04-09T21:41:25.704 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:25.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:25.704 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-09T21:41:25.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:25.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.762+0000 7f0801f6c780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:25.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.762+0000 7f0801f6c780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:25.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:25.766+0000 7f07e36e0700 20 reqs_thread_entry: start 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "32720c28-1ce2-4569-b31f-12fb91c8f57a", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "default.rgw.meta:root", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "default.rgw.control", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "default.rgw.log", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-09T21:41:25.774 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "default.rgw.otp", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "" 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-09T21:41:25.775 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "compression_type": "random" 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "", 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-09T21:41:25.776 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:25.777 DEBUG:tasks.util.rgw: json result: {'id': '32720c28-1ce2-4569-b31f-12fb91c8f57a', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-09T21:41:25.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-04-09T21:41:25.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-04-09T21:41:25.778 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-04-09T21:41:25.853 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:25.854 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:25.864 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:25.861+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.426+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.428 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.426+0000 7f2001c09780 20 realm 2024-04-09T21:41:27.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.426+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.426+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.426+0000 7f2001c09780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-09T21:41:27.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.426+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.430 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.428+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.430 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.428+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.428+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.428+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.429+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.429+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.429+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.429+0000 7f2001c09780 10 Creating default zonegroup 2024-04-09T21:41:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.429+0000 7f2001c09780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-09T21:41:27.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.440+0000 7f2001c09780 10 WARNING: driver->list_zones() returned r=-2 2024-04-09T21:41:27.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.440+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.440+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.443+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.443+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.443+0000 7f2001c09780 10 could not read realm id: (2) No such file or directory 2024-04-09T21:41:27.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.443+0000 7f2001c09780 10 WARNING: failed to set zone as default, r=-22 2024-04-09T21:41:27.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.443+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.443+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.450 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.448+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.448+0000 7f2001c09780 20 rados_obj.operate() r=0 bl.length=874 2024-04-09T21:41:27.451 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.448+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados_obj.operate() r=0 bl.length=420 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 10 Using default name default 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados_obj.operate() r=0 bl.length=46 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados_obj.operate() r=0 bl.length=874 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 zone default found 2024-04-09T21:41:27.452 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 4 Realm: () 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 4 ZoneGroup: default (a7f3ba08-25be-405e-a40c-e1decd59ec7b) 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 4 Zone: default (4d2bb864-458a-4832-8070-08134037ece7) 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 zonegroup default 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:27.453 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:27.449+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:29.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:29.443+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:29.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:29.443+0000 7f2001c09780 20 rados->read ofs=0 len=0 2024-04-09T21:41:29.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:29.445+0000 7f2001c09780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:29.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:29.445+0000 7f2001c09780 20 started sync module instance, tier type = 2024-04-09T21:41:29.448 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:29.445+0000 7f2001c09780 20 started zone id=4d2bb864-458a-4832-8070-08134037ece7 (name=default) with tier type = 2024-04-09T21:41:31.488 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=0 2024-04-09T21:41:31.488 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=1 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=2 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=3 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=4 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=5 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.485+0000 7f2001c09780 20 add_watcher() i=6 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 20 add_watcher() i=7 2024-04-09T21:41:31.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 2 all 8 watchers are set, enabling cache 2024-04-09T21:41:31.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-09T21:41:31.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.487+0000 7f2001c09780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-09T21:41:31.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.488+0000 7f2001c09780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-09T21:41:31.494 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.492+0000 7f2001c09780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-09T21:41:31.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.494+0000 7f1fe6b84700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:31.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.494+0000 7f2001c09780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:31.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:31.494+0000 7f2001c09780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:34.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.443+0000 7f2001c09780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-09T21:41:34.447 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.443+0000 7f2001c09780 5 note: GC not initialized 2024-04-09T21:41:34.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.445+0000 7f2001c09780 20 init_complete bucket index max shards: 11 2024-04-09T21:41:34.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.445+0000 7f2001c09780 10 rgw notify: Started notification manager with: 1 workers 2024-04-09T21:41:34.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.445+0000 7f1fe0377700 20 rgw notify: INFO: next queues processing will happen at: Tue Apr 9 21:42:04 2024 2024-04-09T21:41:34.449 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-04-09T21:41:34.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.446+0000 7f1fdfb55700 20 reqs_thread_entry: start 2024-04-09T21:41:34.449 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.446+0000 7f1fe237b700 20 reqs_thread_entry: start 2024-04-09T21:41:34.469 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.467+0000 7f1fe6b84700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:34.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.469+0000 7f2001c09780 20 remove_watcher() i=0 2024-04-09T21:41:34.471 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.469+0000 7f2001c09780 2 removed watcher, disabling cache 2024-04-09T21:41:34.472 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.470+0000 7f2001c09780 20 remove_watcher() i=1 2024-04-09T21:41:34.473 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.471+0000 7f2001c09780 20 remove_watcher() i=2 2024-04-09T21:41:34.474 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.472+0000 7f2001c09780 20 remove_watcher() i=3 2024-04-09T21:41:34.475 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.473+0000 7f2001c09780 20 remove_watcher() i=4 2024-04-09T21:41:34.476 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.474+0000 7f2001c09780 20 remove_watcher() i=5 2024-04-09T21:41:34.477 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.475+0000 7f2001c09780 20 remove_watcher() i=6 2024-04-09T21:41:34.479 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.477+0000 7f2001c09780 20 remove_watcher() i=7 2024-04-09T21:41:34.482 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2024-04-09T21:41:34.484 DEBUG:tasks.util.rgw: json result: [] 2024-04-09T21:41:34.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:34.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:34.485 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-09T21:41:34.538 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:34.538 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:34.547 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.545+0000 7fd085835780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:34.547 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.545+0000 7fd085835780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:34.551 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.549+0000 7fd066fa9700 20 reqs_thread_entry: start 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "4d2bb864-458a-4832-8070-08134037ece7", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "default.rgw.meta:root", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "default.rgw.control", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "default.rgw.log", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-09T21:41:34.558 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "default.rgw.otp", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "" 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-04-09T21:41:34.559 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:41:34.560 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:41:34.561 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "", 2024-04-09T21:41:34.561 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-09T21:41:34.561 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:41:34.562 DEBUG:tasks.util.rgw: json result: {'id': '4d2bb864-458a-4832-8070-08134037ece7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-09T21:41:34.562 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-04-09T21:41:34.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-04-09T21:41:34.562 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-04-09T21:41:34.605 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:34.605 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:34.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.611+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 realm 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados_obj.operate() r=0 bl.length=46 2024-04-09T21:41:34.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.612+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.613+0000 7f000973e780 20 rados_obj.operate() r=0 bl.length=890 2024-04-09T21:41:34.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.613+0000 7f000973e780 20 searching for the correct realm 2024-04-09T21:41:34.623 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 RGWRados::pool_iterate: got zone_info.4d2bb864-458a-4832-8070-08134037ece7 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 RGWRados::pool_iterate: got zonegroup_info.a7f3ba08-25be-405e-a40c-e1decd59ec7b 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados_obj.operate() r=0 bl.length=46 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados_obj.operate() r=0 bl.length=420 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 zone default found 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 4 Realm: () 2024-04-09T21:41:34.624 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 4 ZoneGroup: default (a7f3ba08-25be-405e-a40c-e1decd59ec7b) 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 4 Zone: default (4d2bb864-458a-4832-8070-08134037ece7) 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 zonegroup default 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.620+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.621+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.621+0000 7f000973e780 20 rados->read ofs=0 len=0 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.621+0000 7f000973e780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.621+0000 7f000973e780 20 started sync module instance, tier type = 2024-04-09T21:41:34.625 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.621+0000 7f000973e780 20 started zone id=4d2bb864-458a-4832-8070-08134037ece7 (name=default) with tier type = 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=0 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=1 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=2 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=3 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=4 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=5 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.630+0000 7f000973e780 20 add_watcher() i=6 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.631+0000 7f000973e780 20 add_watcher() i=7 2024-04-09T21:41:34.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.631+0000 7f000973e780 2 all 8 watchers are set, enabling cache 2024-04-09T21:41:34.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.634+0000 7effee6b9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:34.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.634+0000 7f000973e780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:34.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.634+0000 7f000973e780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:34.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.637+0000 7f000973e780 5 note: GC not initialized 2024-04-09T21:41:34.641 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.638+0000 7effe9eb0700 20 reqs_thread_entry: start 2024-04-09T21:41:34.641 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.638+0000 7f000973e780 20 init_complete bucket index max shards: 11 2024-04-09T21:41:34.641 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.638+0000 7f000973e780 10 rgw notify: Started notification manager with: 1 workers 2024-04-09T21:41:34.641 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.638+0000 7effe7eac700 20 rgw notify: INFO: next queues processing will happen at: Tue Apr 9 21:42:05 2024 2024-04-09T21:41:34.641 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-04-09T21:41:34.641 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.639+0000 7effe768a700 20 reqs_thread_entry: start 2024-04-09T21:41:34.648 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.645+0000 7effee6b9700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-09T21:41:34.652 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.649+0000 7f000973e780 20 remove_watcher() i=0 2024-04-09T21:41:34.652 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.649+0000 7f000973e780 2 removed watcher, disabling cache 2024-04-09T21:41:34.654 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.652+0000 7f000973e780 20 remove_watcher() i=1 2024-04-09T21:41:34.657 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.655+0000 7f000973e780 20 remove_watcher() i=2 2024-04-09T21:41:34.660 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.658+0000 7f000973e780 20 remove_watcher() i=3 2024-04-09T21:41:34.663 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.661+0000 7f000973e780 20 remove_watcher() i=4 2024-04-09T21:41:34.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.663+0000 7f000973e780 20 remove_watcher() i=5 2024-04-09T21:41:34.666 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.664+0000 7f000973e780 20 remove_watcher() i=6 2024-04-09T21:41:34.667 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.665+0000 7f000973e780 20 remove_watcher() i=7 2024-04-09T21:41:34.669 INFO:teuthology.orchestra.run.smithi137.stdout:[] 2024-04-09T21:41:34.671 DEBUG:tasks.util.rgw: json result: [] 2024-04-09T21:41:34.671 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:34.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-09T21:41:34.672 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-09T21:41:34.760 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:34.760 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:34.768 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.766+0000 7f7cdcda3780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-09T21:41:34.769 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.766+0000 7f7cdcda3780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-09T21:41:34.773 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:41:34.770+0000 7f7cbe517700 20 reqs_thread_entry: start 2024-04-09T21:41:34.778 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:41:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "4d2bb864-458a-4832-8070-08134037ece7", 2024-04-09T21:41:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default", 2024-04-09T21:41:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "default.rgw.meta:root", 2024-04-09T21:41:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "default.rgw.control", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "default.rgw.log", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "default.rgw.otp", 2024-04-09T21:41:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "", 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "" 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-09T21:41:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "", 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-09T21:41:34.784 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:41:34.785 DEBUG:tasks.util.rgw: json result: {'id': '4d2bb864-458a-4832-8070-08134037ece7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-09T21:41:34.785 INFO:tasks.rgw:Starting rgw... 2024-04-09T21:41:34.785 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-04-09T21:41:34.785 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-09T21:41:34.786 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-09T21:41:34.786 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-09T21:41:34.788 INFO:tasks.rgw.c1.client.0:Started 2024-04-09T21:41:34.788 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-04-09T21:41:34.788 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-09T21:41:34.789 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-09T21:41:34.789 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-09T21:41:34.791 INFO:tasks.rgw.c1.client.1:Started 2024-04-09T21:41:34.791 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-04-09T21:41:34.791 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-09T21:41:34.791 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-09T21:41:34.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-09T21:41:34.793 INFO:tasks.rgw.c2.client.0:Started 2024-04-09T21:41:34.793 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-04-09T21:41:34.793 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-09T21:41:34.794 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-09T21:41:34.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-09T21:41:34.796 INFO:tasks.rgw.c2.client.1:Started 2024-04-09T21:41:34.796 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi063.front.sepia.ceph.com:8000/ 2024-04-09T21:41:34.796 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000/ 2024-04-09T21:41:34.832 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:34.833 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T21:41:34.834 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8000: Connection refused 2024-04-09T21:41:35.835 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000/ 2024-04-09T21:41:35.858 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:35.858 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:35.859 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:41:35.860 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T21:41:35.860 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi063.front.sepia.ceph.com:8001/ 2024-04-09T21:41:35.861 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001/ 2024-04-09T21:41:35.924 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:35.924 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:35.925 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:41:35.925 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T21:41:35.925 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi137.front.sepia.ceph.com:8000/ 2024-04-09T21:41:35.925 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000/ 2024-04-09T21:41:35.957 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:35.957 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:35.958 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-09T21:41:35.959 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-04-09T21:41:35.959 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi137.front.sepia.ceph.com:8001/ 2024-04-09T21:41:35.959 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001/ 2024-04-09T21:41:36.020 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:36.020 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:36.021 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:41:36.022 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-04-09T21:41:36.022 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-04-09T21:41:37.563 INFO:tasks.rgw_multisite:creating realm.. 2024-04-09T21:41:37.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:37.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:37.564 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:37.617 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:37.617 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:37.650 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:37.651 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:37.651 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "test-realm", 2024-04-09T21:41:37.651 INFO:teuthology.orchestra.run.smithi063.stdout: "current_period": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T21:41:37.651 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1 2024-04-09T21:41:37.651 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:37.652 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-04-09T21:41:37.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:37.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:37.652 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:37.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:37.714+0000 7f8422265780 -1 old format 2024-04-09T21:41:37.730 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T21:41:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T21:41:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T21:41:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T21:41:37.730 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "", 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [], 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [] 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:37.731 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:41:37.732 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:37.733 INFO:tasks.rgw_multisite:creating master zone.. 2024-04-09T21:41:37.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:37.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:37.733 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --endpoints http://smithi063.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:37.830 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:37.831 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:NOTICE: set zone's realm_id=14ac2a00-44c3-4e52-ac2f-2b97df0062c3 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "a1.rgw.control", 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-09T21:41:37.869 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "a1.rgw.log", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "a1.rgw.otp", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:41:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "a1.rgw.buckets.data" 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:37.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:37.872 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-09T21:41:37.872 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:37.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2024-04-09T21:41:38.488 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'a1.rgw.buckets.data' created 2024-04-09T21:41:38.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2024-04-09T21:41:39.491 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2024-04-09T21:41:39.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:39.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:39.513 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:39.568 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:39.568 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "a1.rgw.control", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "a1.rgw.log", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-09T21:41:39.589 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "a1.rgw.otp", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T21:41:39.590 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "a1.rgw.buckets.data", 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "compression_type": "random" 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:39.591 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:39.592 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:39.592 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-09T21:41:39.592 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:39.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:39.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:39.593 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:39.641 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:39.642 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:39.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:39.652+0000 7f60750e5780 0 period (a920d77d-da0e-43d1-9e1e-7c5d1538dcd0 does not have zone a5723776-f142-428c-a6df-d7bf949257da configured 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T21:41:46.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:46.587 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": false, 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T21:41:46.588 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T21:41:46.589 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T21:41:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T21:41:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T21:41:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T21:41:46.593 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:41:46.594 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:46.595 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-09T21:41:46.595 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T21:41:46.596 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T21:41:46.593+0000 7f11acb11700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 46272) UID: 0 2024-04-09T21:41:46.596 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T21:41:46.594+0000 7f11b1ce67c0 -1 shutting down 2024-04-09T21:41:51.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-09T21:41:52.698 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-09T21:41:52.698 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-09T21:41:52.698 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-09T21:41:52.701 INFO:tasks.rgw.c1.client.0:Started 2024-04-09T21:41:52.701 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000 2024-04-09T21:41:52.726 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:52.727 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T21:41:52.728 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8000: Connection refused 2024-04-09T21:41:53.729 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000 2024-04-09T21:41:53.753 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:41:53.753 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:41:53.754 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:41:53.755 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T21:41:53.755 INFO:tasks.rgw_multisite:creating admin user.. 2024-04-09T21:41:53.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:53.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:53.756 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:53.855 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:53.855 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "user_id": "realm-admin", 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "display_name": "Realm Admin", 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "email": "", 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "suspended": 0, 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "max_buckets": 1000, 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "subusers": [], 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: "keys": [ 2024-04-09T21:41:53.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "realm-admin", 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "swift_keys": [], 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": [], 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "op_mask": "read, write, delete", 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "system": true, 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "", 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "default_storage_class": "", 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_tags": [], 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T21:41:53.891 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "temp_url_keys": [], 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "rgw", 2024-04-09T21:41:53.892 INFO:teuthology.orchestra.run.smithi063.stdout: "mfa_ids": [] 2024-04-09T21:41:53.893 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:53.893 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:41:53.902 INFO:tasks.rgw_multisite:creating zone.. 2024-04-09T21:41:53.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:53.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:53.903 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --endpoints http://smithi063.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:53.972 INFO:teuthology.orchestra.run.smithi063.stderr:NOTICE: set zone's realm_id=14ac2a00-44c3-4e52-ac2f-2b97df0062c3 2024-04-09T21:41:53.986 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:53.986 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:41:53.986 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T21:41:53.986 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "a2.rgw.control", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "a2.rgw.log", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "a2.rgw.otp", 2024-04-09T21:41:53.987 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "a2.rgw.buckets.data" 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:53.988 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-09T21:41:53.989 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:53.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2024-04-09T21:41:54.797 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'a2.rgw.buckets.data' created 2024-04-09T21:41:54.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2024-04-09T21:41:55.818 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2024-04-09T21:41:55.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:55.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:55.840 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:55.901 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:55.901 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:55.922 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "a2.rgw.control", 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-09T21:41:55.923 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "a2.rgw.log", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "a2.rgw.otp", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:55.924 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "compression_type": "random" 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:55.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:41:55.926 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:41:55.926 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:41:55.926 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-09T21:41:55.926 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:41:55.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:55.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:41:55.927 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:41:55.975 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:41:55.976 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:41:55.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:41:55.986+0000 7f6211c55780 0 period (feef76df-6795-4193-8fee-aa02d98d3595 does not have zone e208d06e-fe81-4216-93c9-2cb12ee81a65 configured 2024-04-09T21:41:57.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-09T21:42:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone a5723776-f142-428c-a6df-d7bf949257da 2024-04-09T21:42:02.892 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:42:02.892 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:02.892 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 2, 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:02.893 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:02.894 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T21:42:02.895 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:02.896 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T21:42:02.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:42:02.898 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.899 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T21:42:02.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T21:42:02.901 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T21:42:02.912 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:02.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:02.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:02.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T21:42:02.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:02.913 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:42:02.914 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-09T21:42:02.914 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T21:42:02.915 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T21:42:02.913+0000 7fccfdbfd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 46273) UID: 0 2024-04-09T21:42:02.915 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T21:42:02.913+0000 7fcd02dd27c0 -1 shutting down 2024-04-09T21:42:08.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-09T21:42:09.018 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-09T21:42:09.018 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-09T21:42:09.018 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-09T21:42:09.021 INFO:tasks.rgw.c1.client.1:Started 2024-04-09T21:42:09.021 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T21:42:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T21:42:09.050 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:42:09.050 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:42:09.050 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-09T21:42:10.052 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T21:42:10.075 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:42:10.075 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:42:10.075 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:42:10.076 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T21:42:10.077 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-09T21:42:10.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi063.front.sepia.ceph.com:8000', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:10.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi063.front.sepia.ceph.com:8000', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:10.077 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi063.front.sepia.ceph.com:8000 --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:10.135 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:10.135 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:10.183 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:10.184 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:10.184 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-realm", 2024-04-09T21:42:10.184 INFO:teuthology.orchestra.run.smithi137.stdout: "current_period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:10.184 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 2 2024-04-09T21:42:10.184 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:10.185 INFO:tasks.rgw_multisite:creating zonegroup.. 2024-04-09T21:42:10.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:10.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:10.186 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:10.233 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:10.233 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:10.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:42:10.244+0000 7efbf5f3e780 -1 old format 2024-04-09T21:42:10.254 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "b", 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b", 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": false, 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "", 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [], 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [] 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:10.256 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:10.258 INFO:tasks.rgw_multisite:creating zone.. 2024-04-09T21:42:10.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:10.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:10.258 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --endpoints http://smithi137.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:10.351 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:10.352 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:10.365 INFO:teuthology.orchestra.run.smithi137.stderr:NOTICE: set zone's realm_id=14ac2a00-44c3-4e52-ac2f-2b97df0062c3 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b1", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "b1.rgw.control", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "b1.rgw.log", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-09T21:42:10.384 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "b1.rgw.otp", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-04-09T21:42:10.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "b1.rgw.buckets.data" 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-04-09T21:42:10.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:10.387 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:10.387 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:10.387 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:10.387 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-09T21:42:10.387 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:10.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2024-04-09T21:42:11.493 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'b1.rgw.buckets.data' created 2024-04-09T21:42:11.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2024-04-09T21:42:12.516 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2024-04-09T21:42:12.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:12.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:12.537 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:12.589 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:12.590 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:12.609 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:12.609 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:12.609 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b1", 2024-04-09T21:42:12.609 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-09T21:42:12.609 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "b1.rgw.control", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "b1.rgw.log", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "b1.rgw.otp", 2024-04-09T21:42:12.610 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "b1.rgw.buckets.data", 2024-04-09T21:42:12.611 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-09T21:42:12.612 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:12.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:12.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:12.614 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:12.661 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:12.661 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:12.672 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:42:12.670+0000 7f5cfc40c780 0 period (feef76df-6795-4193-8fee-aa02d98d3595 does not have zone 561bc780-8a4a-44a9-b0b2-d59ca12b2689 configured 2024-04-09T21:42:14.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-09T21:42:19.567 INFO:teuthology.orchestra.run.smithi137.stderr:Sending period to new master zone a5723776-f142-428c-a6df-d7bf949257da 2024-04-09T21:42:19.597 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 3, 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": [], 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "period_map": { 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "zonegroups": [ 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a", 2024-04-09T21:42:19.598 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "a", 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": true, 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [ 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "a1", 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:19.599 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "a2", 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:19.601 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [ 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD" 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.602 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "b", 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b", 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": false, 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:19.603 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [ 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b1", 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": false, 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:19.604 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:19.605 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:19.605 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:19.605 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:19.605 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:19.605 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:19.605 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-04-09T21:42:19.610 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [ 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD" 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:19.611 INFO:teuthology.orchestra.run.smithi137.stdout: "short_zone_ids": [ 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 2914326150 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 89109471 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 2084093761 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:19.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zonegroup": "a", 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "period_config": { 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_quota": { 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "user_quota": { 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-04-09T21:42:19.613 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "user_ratelimit": { 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_ratelimit": { 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-04-09T21:42:19.614 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "anonymous_ratelimit": { 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:19.615 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_name": "test-realm", 2024-04-09T21:42:19.616 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:42:19.616 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:19.617 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-09T21:42:19.617 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T21:42:19.618 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-04-09T21:42:19.615+0000 7fbf7f095700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 46167) UID: 0 2024-04-09T21:42:19.618 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-04-09T21:42:19.615+0000 7fbf8426a7c0 -1 shutting down 2024-04-09T21:42:24.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-09T21:42:25.719 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-09T21:42:25.719 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-09T21:42:25.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-09T21:42:25.722 INFO:tasks.rgw.c2.client.0:Started 2024-04-09T21:42:25.722 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-04-09T21:42:25.746 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:42:25.746 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:42:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T21:42:25.747 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8000: Connection refused 2024-04-09T21:42:26.749 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8000 2024-04-09T21:42:26.770 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:42:26.770 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:42:26.771 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:42:26.772 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-04-09T21:42:26.772 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-09T21:42:26.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi063.front.sepia.ceph.com:8000', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--url', 'http://smithi063.front.sepia.ceph.com:8000', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:26.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi063.front.sepia.ceph.com:8000', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--url', 'http://smithi063.front.sepia.ceph.com:8000', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:26.772 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi063.front.sepia.ceph.com:8000 --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --url http://smithi063.front.sepia.ceph.com:8000 --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:26.868 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:26.868 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:26.904 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:26.904 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:26.904 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "test-realm", 2024-04-09T21:42:26.904 INFO:teuthology.orchestra.run.smithi137.stdout: "current_period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:26.904 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 2 2024-04-09T21:42:26.904 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:26.905 INFO:tasks.rgw_multisite:creating zone.. 2024-04-09T21:42:26.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:26.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--endpoints', 'http://smithi137.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:26.906 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --endpoints http://smithi137.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:26.958 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:26.958 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:26.973 INFO:teuthology.orchestra.run.smithi137.stderr:NOTICE: set zone's realm_id=14ac2a00-44c3-4e52-ac2f-2b97df0062c3 2024-04-09T21:42:26.988 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:26.988 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T21:42:26.988 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b2", 2024-04-09T21:42:26.988 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "b2.rgw.control", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "b2.rgw.log", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-09T21:42:26.989 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "b2.rgw.otp", 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:42:26.990 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "b2.rgw.buckets.data" 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:26.991 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:26.992 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-09T21:42:26.992 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:26.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2024-04-09T21:42:27.560 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'b2.rgw.buckets.data' created 2024-04-09T21:42:27.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2024-04-09T21:42:28.565 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2024-04-09T21:42:28.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:28.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:28.584 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:28.639 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:28.639 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:28.657 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:28.657 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T21:42:28.657 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b2", 2024-04-09T21:42:28.657 INFO:teuthology.orchestra.run.smithi137.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-09T21:42:28.657 INFO:teuthology.orchestra.run.smithi137.stdout: "control_pool": "b2.rgw.control", 2024-04-09T21:42:28.657 INFO:teuthology.orchestra.run.smithi137.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "log_pool": "b2.rgw.log", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "otp_pool": "b2.rgw.otp", 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "system_key": { 2024-04-09T21:42:28.658 INFO:teuthology.orchestra.run.smithi137.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_pools": [ 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "default-placement", 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": { 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD": { 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "data_pool": "b2.rgw.buckets.data", 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "compression_type": "random" 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:28.659 INFO:teuthology.orchestra.run.smithi137.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: "index_type": 0, 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: "inline_data": true 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-09T21:42:28.660 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:28.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:28.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:28.661 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:28.709 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:28.709 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:28.721 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:42:28.718+0000 7ff806034780 0 period (feef76df-6795-4193-8fee-aa02d98d3595 does not have zone a98f2146-689b-4f6e-84ee-fe9dc6a42ed3 configured 2024-04-09T21:42:30.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-09T21:42:35.626 INFO:teuthology.orchestra.run.smithi137.stderr:Sending period to new master zone a5723776-f142-428c-a6df-d7bf949257da 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 4, 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": [], 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "period_map": { 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "zonegroups": [ 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a", 2024-04-09T21:42:35.665 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "a", 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": true, 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [ 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "a1", 2024-04-09T21:42:35.666 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:35.667 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "a2", 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:35.668 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [ 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD" 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "b", 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b", 2024-04-09T21:42:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: "api_name": "", 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "is_master": false, 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames": [], 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "hostnames_s3website": [], 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "zones": [ 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b1", 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T21:42:35.671 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:35.672 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "b2", 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "endpoints": [ 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "log_meta": false, 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "log_data": true, 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_index_max_shards": 11, 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "read_only": false, 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_type": "", 2024-04-09T21:42:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from_all": true, 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_from": [], 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "redirect_zone": "", 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "supported_features": [ 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "compress-encrypted", 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "placement_targets": [ 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "default-placement", 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "tags": [], 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "storage_classes": [ 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "STANDARD" 2024-04-09T21:42:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "default_placement": "default-placement", 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_policy": { 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "groups": [] 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled_features": [ 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "resharding" 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "short_zone_ids": [ 2024-04-09T21:42:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 2914326150 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 89109471 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 3383989928 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T21:42:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "val": 2084093761 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zonegroup": "a", 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "period_config": { 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_quota": { 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "user_quota": { 2024-04-09T21:42:35.684 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "check_on_raw": false, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size": -1, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_size_kb": 0, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_objects": -1 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "user_ratelimit": { 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "bucket_ratelimit": { 2024-04-09T21:42:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "anonymous_ratelimit": { 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_ops": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_ops": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_read_bytes": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "max_write_bytes": 0, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "enabled": false 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:42:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T21:42:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_name": "test-realm", 2024-04-09T21:42:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:42:35.687 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:42:35.688 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-09T21:42:35.688 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T21:42:35.689 INFO:tasks.rgw.c2.client.1.smithi137.stdout:2024-04-09T21:42:35.686+0000 7f6c5e0a1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 46168) UID: 0 2024-04-09T21:42:35.689 INFO:tasks.rgw.c2.client.1.smithi137.stdout:2024-04-09T21:42:35.686+0000 7f6c632767c0 -1 shutting down 2024-04-09T21:42:36.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-09T21:42:41.789 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-09T21:42:41.789 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-09T21:42:41.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-09T21:42:41.792 INFO:tasks.rgw.c2.client.1:Started 2024-04-09T21:42:41.792 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001 2024-04-09T21:42:41.815 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:42:41.815 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:42:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T21:42:41.817 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi137.front.sepia.ceph.com port 8001: Connection refused 2024-04-09T21:42:41.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2024-04-09T21:42:42.818 DEBUG:teuthology.orchestra.run.smithi137:> curl http://smithi137.front.sepia.ceph.com:8001 2024-04-09T21:42:42.840 INFO:teuthology.orchestra.run.smithi137.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T21:42:42.840 INFO:teuthology.orchestra.run.smithi137.stderr: Dload Upload Total Spent Left Speed 2024-04-09T21:42:42.840 INFO:teuthology.orchestra.run.smithi137.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T21:42:42.841 INFO:teuthology.orchestra.run.smithi137.stdout:anonymous 2024-04-09T21:42:42.841 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-04-09T21:42:42.842 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-04-09T21:42:42.884 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi 2024-04-09T21:42:43.039 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-04-09T21:42:43.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:43.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T21:42:43.040 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T21:42:43.103 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:43.103 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "user_id": "rgw-multisite-test-user", 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "display_name": "Multisite Test User", 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "email": "", 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "suspended": 0, 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "max_buckets": 1000, 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "subusers": [], 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "keys": [ 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "rgw-multisite-test-user", 2024-04-09T21:42:43.144 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "N5OV6KWZ8WWR3JMWMG4W", 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "6GUgSbTfmuuCFFlIvF01dLwC53sY1XOacwQ5npHN" 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "swift_keys": [], 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": [ 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "roles", 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "perm": "*" 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "op_mask": "read, write, delete", 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "", 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "default_storage_class": "", 2024-04-09T21:42:43.145 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_tags": [], 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T21:42:43.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T21:42:43.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.147 INFO:teuthology.orchestra.run.smithi063.stdout: "temp_url_keys": [], 2024-04-09T21:42:43.147 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "rgw", 2024-04-09T21:42:43.147 INFO:teuthology.orchestra.run.smithi063.stdout: "mfa_ids": [] 2024-04-09T21:42:43.147 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:42:43.147 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-09T21:42:43.157 INFO:rgw_multi.tests:meta checkpoint 2024-04-09T21:42:43.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:43.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:43.157 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:42:43.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:43.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:43.236 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:42:43.247 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:42:43.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.248 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:42:43.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.249 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:42:43.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:42:43.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:43.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:43.250 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:42:43.298 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:43.299 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:43.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:43.326+0000 7f738d933780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:43.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:43.327+0000 7f738d933780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:42:43.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:43.343 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:43.344 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:43.345 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:42:43Z" 2024-04-09T21:42:43.346 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:42:43.347 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:42:47.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-09T21:42:48.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:48.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:48.348 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:42:48.406 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:48.406 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:48.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:48.439+0000 7fbbded59780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:48.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:48.440+0000 7fbbded59780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:48.454 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:42:48.454 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:48.455 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:48.456 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:42:48.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:48.456 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:48.456 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:48.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:48.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:48.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:48.458 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:48.459 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:42:48.460 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:42:48.460 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:42:48.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:48.460 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:42:48Z" 2024-04-09T21:42:48.460 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:42:48.461 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:42:53.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:53.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:53.462 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:42:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:53.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:53.551+0000 7f66041ca780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:53.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:53.552+0000 7f66041ca780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:42:53.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:42:53.568 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:53.569 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:53.570 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:42:53.571 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:42:53.572 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:42:53.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:53.572 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:42:53Z" 2024-04-09T21:42:53.572 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:42:53.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:42:58.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:58.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:42:58.574 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:42:58.632 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:42:58.632 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:42:58.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:58.664+0000 7f915901b780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:58.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:42:58.665+0000 7f915901b780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:58.680 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:58.681 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:42:58.683 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:42:58Z" 2024-04-09T21:42:58.684 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:42:58.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:03.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:03.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:03.687 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:03.745 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:03.745 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:03.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:03.775+0000 7ffb60652780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:03.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:03.776+0000 7ffb60652780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:03.790 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:03.791 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:03.792 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:03.793 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:03.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:03Z" 2024-04-09T21:43:03.795 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:03.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:08.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:08.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:08.797 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:08.854 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:08.854 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:08.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:08.885+0000 7f006394f780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:08.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:08.887+0000 7f006394f780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:08.901 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:08Z" 2024-04-09T21:43:08.906 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:08.907 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:13.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:13.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:13.909 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:13.966 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:13.966 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:14.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:14.000+0000 7f9bbddee780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:14.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:14.001+0000 7f9bbddee780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:14.015 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:14.015 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:14.015 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:14.015 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:14.016 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:14.017 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:14.018 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:14.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:14Z" 2024-04-09T21:43:14.020 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:14.021 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:19.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:19.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:19.023 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:19.080 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:19.081 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:19.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:19.111+0000 7fc9535ca780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:19.112+0000 7fc9535ca780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:19.126 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:19.127 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:19.128 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:19.129 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:19.130 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:19.131 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:19.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:19.131 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:19Z" 2024-04-09T21:43:19.131 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:19.132 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:24.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:24.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:24.133 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:24.190 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:24.191 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:24.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:24.222+0000 7f9874a52780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:24.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:24.223+0000 7f9874a52780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:24.237 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:24.237 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:24.238 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:24.239 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:24.240 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:24Z" 2024-04-09T21:43:24.242 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:24.243 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:29.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:29.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:29.245 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:29.302 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:29.302 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:29.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:29.332+0000 7f1c4a4ee780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:29.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:29.333+0000 7f1c4a4ee780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:29.348 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:29.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:29.352 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:29.353 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:29.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:29.353 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:29Z" 2024-04-09T21:43:29.353 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:29.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:34.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:34.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:34.355 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:34.416 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:34.416 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:34.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:34.446+0000 7f1568eb2780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:34.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:34.447+0000 7f1568eb2780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:34.461 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:34.461 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:34.462 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:34.463 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:34.464 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:34.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:34Z" 2024-04-09T21:43:34.466 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:34.467 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:39.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:39.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:39.468 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:39.527 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:39.528 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:39.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:39.559+0000 7f452f8a0780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:39.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:39.560+0000 7f452f8a0780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:39.574 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:39.576 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:39.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:39.578 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:39Z" 2024-04-09T21:43:39.579 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:39.580 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:44.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:44.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:44.581 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:44.638 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:44.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:44.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:44.670+0000 7f6252e78780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:44.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:44.671+0000 7f6252e78780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:44.686 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:44.686 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:44.686 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:44.686 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:44.686 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:44.687 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:44.688 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:44.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:44.690 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:44.691 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:44.691 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:44.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:44.691 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:44Z" 2024-04-09T21:43:44.691 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:44.692 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:49.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:49.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:49.693 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:49.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:49.781+0000 7f9948011780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:49.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:49.782+0000 7f9948011780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:49.796 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:49.796 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:49.796 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:49.797 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:49.798 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:49.799 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:49.800 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:49Z" 2024-04-09T21:43:49.801 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:49.802 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:54.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:54.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:54.804 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:54.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:54.863 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:43:54.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:54.894+0000 7f60734fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:54.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:43:54.895+0000 7f60734fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:43:54.910 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:43:54.910 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:43:54.910 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:43:54.910 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:43:54.911 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:54.912 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:54.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:43:54.914 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:43:54Z" 2024-04-09T21:43:54.915 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:43:54.916 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:43:59.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:59.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:43:59.917 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:43:59.978 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:43:59.978 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:00.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:00.008+0000 7f7d9c0fb780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:00.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:00.009+0000 7f7d9c0fb780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:00Z" 2024-04-09T21:44:00.029 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:00.030 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:05.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:05.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:05.031 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:05.088 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:05.088 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:05.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:05.120+0000 7f9f3ffdc780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:05.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:05.121+0000 7f9f3ffdc780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:05.135 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:05.135 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:05.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:05Z" 2024-04-09T21:44:05.140 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:05.141 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:10.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:10.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:10.142 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:10.200 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:10.200 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:10.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:10.231+0000 7ff022c79780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:10.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:10.232+0000 7ff022c79780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:10.247 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:10.248 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:10.249 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:10.250 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:10.251 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:10Z" 2024-04-09T21:44:10.252 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:10.254 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:15.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:15.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:15.255 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:15.312 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:15.312 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:15.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:15.344+0000 7f39be2ea780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:15.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:15.345+0000 7f39be2ea780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:15.360 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:15.361 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:15.362 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:15.363 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:15Z" 2024-04-09T21:44:15.364 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:15.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:20.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:20.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:20.367 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:20.426 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:20.426 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:20.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:20.458+0000 7fd441781780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:20.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:20.459+0000 7fd441781780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:20.473 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:20.473 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:20.473 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:20.473 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:20.473 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:20.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:20.475 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:20Z" 2024-04-09T21:44:20.478 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:20.479 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:25.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:25.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:25.480 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:25.540 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:25.540 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:25.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:25.573+0000 7f5e1ac48780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:25.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:25.574+0000 7f5e1ac48780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:25.588 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:25.590 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:25.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:25.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:25Z" 2024-04-09T21:44:25.593 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:25.594 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:30.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:30.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:30.596 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:30.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:30.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:30.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:30.684+0000 7f3160350780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:30.686+0000 7f3160350780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:30.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:30Z" 2024-04-09T21:44:30.704 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:30.705 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:35.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:35.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:35.706 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:35.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:35.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:35.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:35.795+0000 7f37d2259780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:35.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:35.796+0000 7f37d2259780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:35.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:35.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:35.812 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:35.813 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:35Z" 2024-04-09T21:44:35.814 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:35.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712698963.140128_76.1 target= 2024-04-09T21:44:40.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:40.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:40.817 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:40.905+0000 7fb520d46780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:40.906+0000 7fb520d46780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:40.924 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:40Z" 2024-04-09T21:44:40.925 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:40.926 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:40.926 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-09T21:44:40.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:40.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:40.927 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:40.982 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:40.983 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:41.019 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.018+0000 7f899991c780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.020 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.020+0000 7f899991c780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.032 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:44:41.032 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-04-09T21:44:41.032 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-04-09T21:44:41.032 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-04-09T21:44:41.032 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-04-09T21:44:41.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.034 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.036 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-04-09T21:44:41Z" 2024-04-09T21:44:41.037 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:44:41.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-09T21:44:41.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-09T21:44:41.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.039 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:41.088 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:41.088 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:41.118 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.118+0000 7fccfbd03780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.119+0000 7fccfbd03780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.130 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:44:41.130 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-04-09T21:44:41.130 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-04-09T21:44:41.130 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-04-09T21:44:41.130 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:41.131 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-04-09T21:44:41.133 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-04-09T21:44:41.134 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-04-09T21:44:41.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.135 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-04-09T21:44:41Z" 2024-04-09T21:44:41.135 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:44:41.136 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-09T21:44:41.136 INFO:rgw_multi.tests:meta checkpoint 2024-04-09T21:44:41.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.136 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:41.188 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:41.188 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:41.221 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:41.235 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:41.236 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:41.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.237 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:41.286 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:41.286 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:41.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:41.313+0000 7fe4089dc780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:41.314+0000 7fe4089dc780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:41.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:41.329 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:41.330 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:41.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:41Z" 2024-04-09T21:44:41.332 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:41.333 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:41.333 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-09T21:44:41.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.333 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:41.381 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:41.382 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:41.408 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.407+0000 7f66001f7780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.409 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.408+0000 7f66001f7780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.421 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:44:41.421 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-04-09T21:44:41.421 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-04-09T21:44:41.421 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-04-09T21:44:41.421 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-04-09T21:44:41.421 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.422 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.423 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.424 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.425 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-04-09T21:44:41Z" 2024-04-09T21:44:41.426 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:44:41.426 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-09T21:44:41.427 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-09T21:44:41.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:41.427 DEBUG:teuthology.orchestra.run.smithi137:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:41.476 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:41.476 INFO:teuthology.orchestra.run.smithi137.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:41.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.506+0000 7f7c8a9ee780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-04-09T21:44:41.507+0000 7f7c8a9ee780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:41.519 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "sync_status": { 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "info": { 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "status": "sync", 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "num_shards": 4, 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "markers": [ 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 0, 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:41.520 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 1, 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 1, 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.521 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 2, 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.522 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "key": 3, 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "val": { 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "state": 1, 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "marker": "", 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "next_step_marker": "", 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "total_entries": 0, 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "pos": 0, 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "timestamp": "0.000000", 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: "realm_epoch": 2 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-04-09T21:44:41.523 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout: "full_sync": { 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout: "total": 1, 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout: "complete": 1 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout: "current_time": "2024-04-09T21:44:41Z" 2024-04-09T21:44:41.524 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-04-09T21:44:41.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-09T21:44:41.525 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-04-09T21:44:41.916 DEBUG:boto:Using access key provided by client. 2024-04-09T21:44:41.916 DEBUG:boto:Using secret key provided by client. 2024-04-09T21:44:41.917 DEBUG:boto:Using access key provided by client. 2024-04-09T21:44:41.917 DEBUG:boto:Using secret key provided by client. 2024-04-09T21:44:41.917 DEBUG:boto:Using access key provided by client. 2024-04-09T21:44:41.917 DEBUG:boto:Using secret key provided by client. 2024-04-09T21:44:41.918 DEBUG:boto:Using access key provided by client. 2024-04-09T21:44:41.918 DEBUG:boto:Using secret key provided by client. 2024-04-09T21:44:41.918 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-1 2024-04-09T21:44:41.918 DEBUG:boto:path=/xryqqk-1/ 2024-04-09T21:44:41.918 DEBUG:boto:auth_path=/xryqqk-1/ 2024-04-09T21:44:41.919 DEBUG:boto:Method: PUT 2024-04-09T21:44:41.919 DEBUG:boto:Path: /xryqqk-1/ 2024-04-09T21:44:41.919 DEBUG:boto:Data: 2024-04-09T21:44:41.919 DEBUG:boto:Headers: {} 2024-04-09T21:44:41.919 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:41.919 DEBUG:boto:Port: 8000 2024-04-09T21:44:41.919 DEBUG:boto:Params: {} 2024-04-09T21:44:41.919 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T21:44:41.919 DEBUG:boto:Token: None 2024-04-09T21:44:41.919 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:41 GMT /xryqqk-1/ 2024-04-09T21:44:41.920 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PGO9BEnXTwOobbrJlNyPjRlbOG4= 2024-04-09T21:44:41.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PGO9BEnXTwOobbrJlNyPjRlbOG4=', 'Content-Length': '0'} 2024-04-09T21:44:44.071 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000077abd91787c58a49-006615b6c9-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:44.071 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-2 2024-04-09T21:44:44.071 DEBUG:boto:path=/xryqqk-2/ 2024-04-09T21:44:44.071 DEBUG:boto:auth_path=/xryqqk-2/ 2024-04-09T21:44:44.071 DEBUG:boto:Method: PUT 2024-04-09T21:44:44.072 DEBUG:boto:Path: /xryqqk-2/ 2024-04-09T21:44:44.072 DEBUG:boto:Data: 2024-04-09T21:44:44.072 DEBUG:boto:Headers: {} 2024-04-09T21:44:44.072 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:44.072 DEBUG:boto:Port: 8001 2024-04-09T21:44:44.072 DEBUG:boto:Params: {} 2024-04-09T21:44:44.072 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T21:44:44.072 DEBUG:boto:Token: None 2024-04-09T21:44:44.073 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:44 GMT /xryqqk-2/ 2024-04-09T21:44:44.073 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CbZB7cxOpq2tEjfYUJRfOFp5g6I= 2024-04-09T21:44:44.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CbZB7cxOpq2tEjfYUJRfOFp5g6I=', 'Content-Length': '0'} 2024-04-09T21:44:47.070 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000064653006ccd4e499-006615b6cc-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:47.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:47.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:47.071 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:47.122 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:47.122 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:47.153 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:47.166 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:47.166 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:47.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.168 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699084.088872_43.1", 2024-04-09T21:44:47.168 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:44:44.088872Z" 2024-04-09T21:44:47.168 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:47.168 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:47.169 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699084.088872_43.1'} 2024-04-09T21:44:47.169 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:47.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:47.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:47.170 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:47.218 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:47.219 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:47.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:47.246+0000 7f323cb14780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:47.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:47.247+0000 7f323cb14780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.260 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:47.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:47.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:47.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699084.058835_36.1", 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:44.058835Z", 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:47.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:47Z" 2024-04-09T21:44:47.265 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:47.266 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699084.058835_36.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:44.058835Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:47Z" } 2024-04-09T21:44:47.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699084.058835_36.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:44.058835Z', 'realm_epoch': 2}}] 2024-04-09T21:44:47.267 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699084.088872_43.1'} 2024-04-09T21:44:47.267 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699084.058835_36.1'} 2024-04-09T21:44:47.267 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712699084.088872_43.1 target=1_1712699084.058835_36.1 2024-04-09T21:44:52.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.268 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:52.326 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:52.326 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:52.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:52.360+0000 7f6024898780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:52.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:52.361+0000 7f6024898780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:52.378 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.379 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.380 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.381 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699084.088872_43.1", 2024-04-09T21:44:52.382 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:44.088872Z", 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:52.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.384 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:52Z" 2024-04-09T21:44:52.384 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:52.384 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699084.088872_43.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:44.088872Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:52Z" } 2024-04-09T21:44:52.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699084.088872_43.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:44.088872Z', 'realm_epoch': 2}}] 2024-04-09T21:44:52.385 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699084.088872_43.1'} 2024-04-09T21:44:52.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699084.088872_43.1'} 2024-04-09T21:44:52.385 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:52.385 DEBUG:boto:path=/xryqqk-1/ 2024-04-09T21:44:52.385 DEBUG:boto:auth_path=/xryqqk-1/ 2024-04-09T21:44:52.385 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.385 DEBUG:boto:Path: /xryqqk-1/ 2024-04-09T21:44:52.385 DEBUG:boto:Data: 2024-04-09T21:44:52.386 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.386 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.386 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.386 DEBUG:boto:Params: {} 2024-04-09T21:44:52.386 DEBUG:boto:Token: None 2024-04-09T21:44:52.386 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-1/ 2024-04-09T21:44:52.386 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mZo0sPOuRPd+Q2PFopKrnwQLD9Y= 2024-04-09T21:44:52.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mZo0sPOuRPd+Q2PFopKrnwQLD9Y=', 'Content-Length': '0'} 2024-04-09T21:44:52.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f8770000b781aa4-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.389 DEBUG:boto:path=/xryqqk-2/ 2024-04-09T21:44:52.389 DEBUG:boto:auth_path=/xryqqk-2/ 2024-04-09T21:44:52.389 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.390 DEBUG:boto:Path: /xryqqk-2/ 2024-04-09T21:44:52.390 DEBUG:boto:Data: 2024-04-09T21:44:52.390 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.390 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.390 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.390 DEBUG:boto:Params: {} 2024-04-09T21:44:52.390 DEBUG:boto:Token: None 2024-04-09T21:44:52.390 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-2/ 2024-04-09T21:44:52.390 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UgrWYzaNfcuD0+loSZQ3kHDMRfM= 2024-04-09T21:44:52.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UgrWYzaNfcuD0+loSZQ3kHDMRfM=', 'Content-Length': '0'} 2024-04-09T21:44:52.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000026edd4a2f48f8b58-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.393 DEBUG:boto:path=/xryqqk-1/ 2024-04-09T21:44:52.393 DEBUG:boto:auth_path=/xryqqk-1/ 2024-04-09T21:44:52.393 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.393 DEBUG:boto:Path: /xryqqk-1/ 2024-04-09T21:44:52.393 DEBUG:boto:Data: 2024-04-09T21:44:52.393 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.393 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.393 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.393 DEBUG:boto:Params: {} 2024-04-09T21:44:52.393 DEBUG:boto:Token: None 2024-04-09T21:44:52.394 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-1/ 2024-04-09T21:44:52.394 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mZo0sPOuRPd+Q2PFopKrnwQLD9Y= 2024-04-09T21:44:52.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mZo0sPOuRPd+Q2PFopKrnwQLD9Y=', 'Content-Length': '0'} 2024-04-09T21:44:52.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4f91f48cb093384-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.396 DEBUG:boto:path=/xryqqk-2/ 2024-04-09T21:44:52.396 DEBUG:boto:auth_path=/xryqqk-2/ 2024-04-09T21:44:52.396 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.397 DEBUG:boto:Path: /xryqqk-2/ 2024-04-09T21:44:52.397 DEBUG:boto:Data: 2024-04-09T21:44:52.397 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.397 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.397 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.397 DEBUG:boto:Params: {} 2024-04-09T21:44:52.397 DEBUG:boto:Token: None 2024-04-09T21:44:52.397 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-2/ 2024-04-09T21:44:52.397 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UgrWYzaNfcuD0+loSZQ3kHDMRfM= 2024-04-09T21:44:52.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UgrWYzaNfcuD0+loSZQ3kHDMRfM=', 'Content-Length': '0'} 2024-04-09T21:44:52.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc682d37c54388a2-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.400 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-04-09T21:44:52.400 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-3 2024-04-09T21:44:52.401 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.401 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.401 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.401 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.401 DEBUG:boto:Data: 2024-04-09T21:44:52.401 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.401 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.401 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.401 DEBUG:boto:Params: {} 2024-04-09T21:44:52.401 DEBUG:boto:Token: None 2024-04-09T21:44:52.402 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.402 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:m3FRfBxrxSxMTed/1/EZMOnKc08= 2024-04-09T21:44:52.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:m3FRfBxrxSxMTed/1/EZMOnKc08=', 'Content-Length': '0'} 2024-04-09T21:44:52.415 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6623532d99eeff3-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.415 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-4 2024-04-09T21:44:52.415 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.415 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.415 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.415 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.415 DEBUG:boto:Data: 2024-04-09T21:44:52.415 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.416 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.416 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.416 DEBUG:boto:Params: {} 2024-04-09T21:44:52.416 DEBUG:boto:Token: None 2024-04-09T21:44:52.416 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.416 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GfMqscRsZMLCZwQMSSKBJtCk7TM= 2024-04-09T21:44:52.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GfMqscRsZMLCZwQMSSKBJtCk7TM=', 'Content-Length': '0'} 2024-04-09T21:44:52.444 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000042a31765381ebd24-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.445 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:52.496 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:52.497 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:52.533 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.596 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699092.430618_51.1", 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:44:52.430618Z" 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.597 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:52.598 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699092.430618_51.1'} 2024-04-09T21:44:52.598 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:52.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.598 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:52.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:52.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:52.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:52.676+0000 7f383c17a780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:52.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:52.677+0000 7f383c17a780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:52.689 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:52.689 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:52.689 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:52.689 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.690 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.691 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.692 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.693 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699092.430618_51.1", 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:52.430618Z", 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.694 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:52.695 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:52.695 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:52.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.695 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:52Z" 2024-04-09T21:44:52.695 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:52.696 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699092.430618_51.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:52.430618Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:52Z" } 2024-04-09T21:44:52.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699092.430618_51.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:52.430618Z', 'realm_epoch': 2}}] 2024-04-09T21:44:52.696 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699092.430618_51.1'} 2024-04-09T21:44:52.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699092.430618_51.1'} 2024-04-09T21:44:52.696 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:52.697 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.697 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.697 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.697 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.697 DEBUG:boto:Data: 2024-04-09T21:44:52.697 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.697 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.697 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.697 DEBUG:boto:Params: {} 2024-04-09T21:44:52.697 DEBUG:boto:Token: None 2024-04-09T21:44:52.698 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.698 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM= 2024-04-09T21:44:52.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM=', 'Content-Length': '0'} 2024-04-09T21:44:52.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093b0334ddb322194-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.700 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.700 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.700 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.700 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.700 DEBUG:boto:Data: 2024-04-09T21:44:52.700 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.701 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.701 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.701 DEBUG:boto:Params: {} 2024-04-09T21:44:52.701 DEBUG:boto:Token: None 2024-04-09T21:44:52.701 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.701 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA= 2024-04-09T21:44:52.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA=', 'Content-Length': '0'} 2024-04-09T21:44:52.703 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002558bc0933df9f34-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.703 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.703 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.703 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.703 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.703 DEBUG:boto:Data: 2024-04-09T21:44:52.704 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.704 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.704 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.704 DEBUG:boto:Params: {} 2024-04-09T21:44:52.704 DEBUG:boto:Token: None 2024-04-09T21:44:52.704 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.704 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM= 2024-04-09T21:44:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM=', 'Content-Length': '0'} 2024-04-09T21:44:52.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000424c70004769aa99-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.706 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.706 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.706 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.706 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.707 DEBUG:boto:Data: 2024-04-09T21:44:52.707 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.707 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.707 DEBUG:boto:Params: {} 2024-04-09T21:44:52.707 DEBUG:boto:Token: None 2024-04-09T21:44:52.707 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.707 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA= 2024-04-09T21:44:52.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA=', 'Content-Length': '0'} 2024-04-09T21:44:52.709 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b3a9511f937fcd7-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.709 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.709 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.709 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.710 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.710 DEBUG:boto:Data: 2024-04-09T21:44:52.710 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.710 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.710 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.710 DEBUG:boto:Params: {} 2024-04-09T21:44:52.710 DEBUG:boto:Token: None 2024-04-09T21:44:52.710 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.710 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:m3FRfBxrxSxMTed/1/EZMOnKc08= 2024-04-09T21:44:52.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:m3FRfBxrxSxMTed/1/EZMOnKc08=', 'Content-Length': '0'} 2024-04-09T21:44:52.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007367266fa47de958-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.713 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.714 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.714 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.714 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.714 DEBUG:boto:Data: 2024-04-09T21:44:52.714 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.714 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.714 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.714 DEBUG:boto:Params: {} 2024-04-09T21:44:52.714 DEBUG:boto:Token: None 2024-04-09T21:44:52.714 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.715 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GfMqscRsZMLCZwQMSSKBJtCk7TM= 2024-04-09T21:44:52.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GfMqscRsZMLCZwQMSSKBJtCk7TM=', 'Content-Length': '0'} 2024-04-09T21:44:52.717 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dc3af7d5d5e1983f-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.717 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.717 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.717 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.717 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.717 DEBUG:boto:Data: 2024-04-09T21:44:52.717 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.718 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.718 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.718 DEBUG:boto:Params: {} 2024-04-09T21:44:52.718 DEBUG:boto:Token: None 2024-04-09T21:44:52.718 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.718 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:m3FRfBxrxSxMTed/1/EZMOnKc08= 2024-04-09T21:44:52.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:m3FRfBxrxSxMTed/1/EZMOnKc08=', 'Content-Length': '0'} 2024-04-09T21:44:52.722 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eb5eda5f132d8006-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.723 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.723 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.723 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.723 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.723 DEBUG:boto:Data: 2024-04-09T21:44:52.723 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.723 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.723 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.723 DEBUG:boto:Params: {} 2024-04-09T21:44:52.723 DEBUG:boto:Token: None 2024-04-09T21:44:52.724 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.724 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GfMqscRsZMLCZwQMSSKBJtCk7TM= 2024-04-09T21:44:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GfMqscRsZMLCZwQMSSKBJtCk7TM=', 'Content-Length': '0'} 2024-04-09T21:44:52.769 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b68c2525fb87f76b-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.769 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.769 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.769 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.769 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.769 DEBUG:boto:Data: 2024-04-09T21:44:52.770 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.770 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.770 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.770 DEBUG:boto:Params: {} 2024-04-09T21:44:52.770 DEBUG:boto:Token: None 2024-04-09T21:44:52.770 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.770 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM= 2024-04-09T21:44:52.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM=', 'Content-Length': '0'} 2024-04-09T21:44:52.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000795a8c5c6a02304-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.772 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.772 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.773 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.773 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.773 DEBUG:boto:Data: 2024-04-09T21:44:52.773 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.773 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.773 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.773 DEBUG:boto:Params: {} 2024-04-09T21:44:52.773 DEBUG:boto:Token: None 2024-04-09T21:44:52.773 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.774 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA= 2024-04-09T21:44:52.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA=', 'Content-Length': '0'} 2024-04-09T21:44:52.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eff4db3228f5bab8-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.775 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.776 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.776 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.776 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.776 DEBUG:boto:Data: 2024-04-09T21:44:52.776 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.776 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.776 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.776 DEBUG:boto:Params: {} 2024-04-09T21:44:52.776 DEBUG:boto:Token: None 2024-04-09T21:44:52.777 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.777 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM= 2024-04-09T21:44:52.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM=', 'Content-Length': '0'} 2024-04-09T21:44:52.778 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009493794fe9b76511-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.779 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.779 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.779 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.779 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.779 DEBUG:boto:Data: 2024-04-09T21:44:52.779 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.779 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.779 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.779 DEBUG:boto:Params: {} 2024-04-09T21:44:52.779 DEBUG:boto:Token: None 2024-04-09T21:44:52.780 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA= 2024-04-09T21:44:52.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA=', 'Content-Length': '0'} 2024-04-09T21:44:52.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075d3c3526c2b6e73-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.782 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:52.830 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:52.831 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:52.860 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:52.872 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:52.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:52.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.874 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699092.430618_51.1", 2024-04-09T21:44:52.874 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:44:52.430618Z" 2024-04-09T21:44:52.874 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.874 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:52.875 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699092.430618_51.1'} 2024-04-09T21:44:52.875 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:52.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:52.876 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:52.924 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:52.924 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:52.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:52.952+0000 7fd8bf1ff780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:52.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:52.953+0000 7fd8bf1ff780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:52.965 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:52.965 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:52.965 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:52.965 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:52.965 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:52.965 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:52.966 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.967 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.968 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699092.430618_51.1", 2024-04-09T21:44:52.969 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:52.430618Z", 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:52.970 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:52.971 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:52.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:52.971 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:52Z" 2024-04-09T21:44:52.971 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:52.972 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699092.430618_51.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:52.430618Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:52Z" } 2024-04-09T21:44:52.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699092.430618_51.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:52.430618Z', 'realm_epoch': 2}}] 2024-04-09T21:44:52.972 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699092.430618_51.1'} 2024-04-09T21:44:52.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699092.430618_51.1'} 2024-04-09T21:44:52.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:52.973 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.973 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.973 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.973 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.973 DEBUG:boto:Data: 2024-04-09T21:44:52.973 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.973 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.973 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.973 DEBUG:boto:Params: {} 2024-04-09T21:44:52.974 DEBUG:boto:Token: None 2024-04-09T21:44:52.974 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.974 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM= 2024-04-09T21:44:52.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM=', 'Content-Length': '0'} 2024-04-09T21:44:52.976 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000648c031ab6112489-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.976 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.976 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.976 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.976 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.977 DEBUG:boto:Data: 2024-04-09T21:44:52.977 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.977 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.977 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.977 DEBUG:boto:Params: {} 2024-04-09T21:44:52.977 DEBUG:boto:Token: None 2024-04-09T21:44:52.977 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.977 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA= 2024-04-09T21:44:52.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA=', 'Content-Length': '0'} 2024-04-09T21:44:52.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023909c34c7bcac73-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.979 DEBUG:boto:path=/xryqqk-3/ 2024-04-09T21:44:52.979 DEBUG:boto:auth_path=/xryqqk-3/ 2024-04-09T21:44:52.979 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.979 DEBUG:boto:Path: /xryqqk-3/ 2024-04-09T21:44:52.979 DEBUG:boto:Data: 2024-04-09T21:44:52.980 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.980 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.980 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.980 DEBUG:boto:Params: {} 2024-04-09T21:44:52.980 DEBUG:boto:Token: None 2024-04-09T21:44:52.980 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-3/ 2024-04-09T21:44:52.980 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM= 2024-04-09T21:44:52.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLEQO05SQNYwDT+quQo7y+OZLEM=', 'Content-Length': '0'} 2024-04-09T21:44:52.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020f198a0a6f0e4fb-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.982 DEBUG:boto:path=/xryqqk-4/ 2024-04-09T21:44:52.982 DEBUG:boto:auth_path=/xryqqk-4/ 2024-04-09T21:44:52.983 DEBUG:boto:Method: HEAD 2024-04-09T21:44:52.983 DEBUG:boto:Path: /xryqqk-4/ 2024-04-09T21:44:52.983 DEBUG:boto:Data: 2024-04-09T21:44:52.983 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.983 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:52.983 DEBUG:boto:Port: 8001 2024-04-09T21:44:52.983 DEBUG:boto:Params: {} 2024-04-09T21:44:52.983 DEBUG:boto:Token: None 2024-04-09T21:44:52.983 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-4/ 2024-04-09T21:44:52.983 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA= 2024-04-09T21:44:52.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7fJpfGmPHwJgVvtMNF+jbjHr4SA=', 'Content-Length': '0'} 2024-04-09T21:44:52.985 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2270835787f8a7c-006615b6d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:52.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-04-09T21:44:52.986 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-5 2024-04-09T21:44:52.986 DEBUG:boto:path=/xryqqk-5/ 2024-04-09T21:44:52.986 DEBUG:boto:auth_path=/xryqqk-5/ 2024-04-09T21:44:52.986 DEBUG:boto:Method: PUT 2024-04-09T21:44:52.986 DEBUG:boto:Path: /xryqqk-5/ 2024-04-09T21:44:52.986 DEBUG:boto:Data: 2024-04-09T21:44:52.987 DEBUG:boto:Headers: {} 2024-04-09T21:44:52.987 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:52.987 DEBUG:boto:Port: 8000 2024-04-09T21:44:52.987 DEBUG:boto:Params: {} 2024-04-09T21:44:52.987 DEBUG:boto:Token: None 2024-04-09T21:44:52.987 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:52 GMT /xryqqk-5/ 2024-04-09T21:44:52.987 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:eRuCAXTq6ur6SABKZDPEVmG2OO4= 2024-04-09T21:44:52.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eRuCAXTq6ur6SABKZDPEVmG2OO4=', 'Content-Length': '0'} 2024-04-09T21:44:53.000 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001d1974266fcba620-006615b6d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.000 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-6 2024-04-09T21:44:53.000 DEBUG:boto:path=/xryqqk-6/ 2024-04-09T21:44:53.000 DEBUG:boto:auth_path=/xryqqk-6/ 2024-04-09T21:44:53.000 DEBUG:boto:Method: PUT 2024-04-09T21:44:53.000 DEBUG:boto:Path: /xryqqk-6/ 2024-04-09T21:44:53.001 DEBUG:boto:Data: 2024-04-09T21:44:53.001 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.001 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:53.001 DEBUG:boto:Port: 8001 2024-04-09T21:44:53.001 DEBUG:boto:Params: {} 2024-04-09T21:44:53.001 DEBUG:boto:Token: None 2024-04-09T21:44:53.001 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-6/ 2024-04-09T21:44:53.001 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jQxrMADzJiDgixn+TTKtdGIKaOg= 2024-04-09T21:44:53.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jQxrMADzJiDgixn+TTKtdGIKaOg=', 'Content-Length': '0'} 2024-04-09T21:44:53.023 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000002dc25f8fb11a98f-006615b6d5-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.024 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:53.117 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:53.129 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:53.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.129 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:53.129 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:53.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699093.013144_59.1", 2024-04-09T21:44:53.130 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:44:53.013144Z" 2024-04-09T21:44:53.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.131 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:53.132 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.013144_59.1'} 2024-04-09T21:44:53.132 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:53.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.132 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:53.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:53.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:53.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:53.209+0000 7fa1cae09780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:53.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:53.210+0000 7fa1cae09780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:53.222 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:53.222 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.223 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.224 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.225 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699093.013144_59.1", 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:53.013144Z", 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:53.227 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:53.228 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:53.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.228 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:53Z" 2024-04-09T21:44:53.228 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:53.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699093.013144_59.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:53.013144Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:53Z" } 2024-04-09T21:44:53.229 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699093.013144_59.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:53.013144Z', 'realm_epoch': 2}}] 2024-04-09T21:44:53.229 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.013144_59.1'} 2024-04-09T21:44:53.229 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.013144_59.1'} 2024-04-09T21:44:53.229 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:53.229 DEBUG:boto:path=/xryqqk-5/ 2024-04-09T21:44:53.229 DEBUG:boto:auth_path=/xryqqk-5/ 2024-04-09T21:44:53.230 DEBUG:boto:Method: HEAD 2024-04-09T21:44:53.230 DEBUG:boto:Path: /xryqqk-5/ 2024-04-09T21:44:53.230 DEBUG:boto:Data: 2024-04-09T21:44:53.230 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.230 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:53.230 DEBUG:boto:Port: 8000 2024-04-09T21:44:53.230 DEBUG:boto:Params: {} 2024-04-09T21:44:53.230 DEBUG:boto:Token: None 2024-04-09T21:44:53.230 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-5/ 2024-04-09T21:44:53.231 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1c30qjaL9EAh9QJIDlY8qowWcjs= 2024-04-09T21:44:53.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1c30qjaL9EAh9QJIDlY8qowWcjs=', 'Content-Length': '0'} 2024-04-09T21:44:53.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000392d20d21ed7f24b-006615b6d5-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.233 DEBUG:boto:path=/xryqqk-6/ 2024-04-09T21:44:53.233 DEBUG:boto:auth_path=/xryqqk-6/ 2024-04-09T21:44:53.233 DEBUG:boto:Method: HEAD 2024-04-09T21:44:53.233 DEBUG:boto:Path: /xryqqk-6/ 2024-04-09T21:44:53.233 DEBUG:boto:Data: 2024-04-09T21:44:53.233 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.233 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:53.234 DEBUG:boto:Port: 8000 2024-04-09T21:44:53.234 DEBUG:boto:Params: {} 2024-04-09T21:44:53.234 DEBUG:boto:Token: None 2024-04-09T21:44:53.234 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-6/ 2024-04-09T21:44:53.234 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:415eXIJDKtRgzGfHKLweOlQ4w6E= 2024-04-09T21:44:53.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:415eXIJDKtRgzGfHKLweOlQ4w6E=', 'Content-Length': '0'} 2024-04-09T21:44:53.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a6096440cd2a58c-006615b6d5-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.236 DEBUG:boto:path=/xryqqk-5/ 2024-04-09T21:44:53.236 DEBUG:boto:auth_path=/xryqqk-5/ 2024-04-09T21:44:53.236 DEBUG:boto:Method: HEAD 2024-04-09T21:44:53.236 DEBUG:boto:Path: /xryqqk-5/ 2024-04-09T21:44:53.236 DEBUG:boto:Data: 2024-04-09T21:44:53.236 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.236 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:53.236 DEBUG:boto:Port: 8001 2024-04-09T21:44:53.237 DEBUG:boto:Params: {} 2024-04-09T21:44:53.237 DEBUG:boto:Token: None 2024-04-09T21:44:53.237 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-5/ 2024-04-09T21:44:53.237 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1c30qjaL9EAh9QJIDlY8qowWcjs= 2024-04-09T21:44:53.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1c30qjaL9EAh9QJIDlY8qowWcjs=', 'Content-Length': '0'} 2024-04-09T21:44:53.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008466a879cde532f8-006615b6d5-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.239 DEBUG:boto:path=/xryqqk-6/ 2024-04-09T21:44:53.239 DEBUG:boto:auth_path=/xryqqk-6/ 2024-04-09T21:44:53.239 DEBUG:boto:Method: HEAD 2024-04-09T21:44:53.239 DEBUG:boto:Path: /xryqqk-6/ 2024-04-09T21:44:53.239 DEBUG:boto:Data: 2024-04-09T21:44:53.239 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.240 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:53.240 DEBUG:boto:Port: 8001 2024-04-09T21:44:53.240 DEBUG:boto:Params: {} 2024-04-09T21:44:53.240 DEBUG:boto:Token: None 2024-04-09T21:44:53.240 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-6/ 2024-04-09T21:44:53.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:415eXIJDKtRgzGfHKLweOlQ4w6E= 2024-04-09T21:44:53.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:415eXIJDKtRgzGfHKLweOlQ4w6E=', 'Content-Length': '0'} 2024-04-09T21:44:53.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e29273f86fb99e9-006615b6d5-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.242 DEBUG:boto:path=/xryqqk-5/ 2024-04-09T21:44:53.242 DEBUG:boto:auth_path=/xryqqk-5/ 2024-04-09T21:44:53.242 DEBUG:boto:Method: DELETE 2024-04-09T21:44:53.243 DEBUG:boto:Path: /xryqqk-5/ 2024-04-09T21:44:53.243 DEBUG:boto:Data: 2024-04-09T21:44:53.243 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.243 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:53.243 DEBUG:boto:Port: 8000 2024-04-09T21:44:53.243 DEBUG:boto:Params: {} 2024-04-09T21:44:53.243 DEBUG:boto:Token: None 2024-04-09T21:44:53.243 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-5/ 2024-04-09T21:44:53.243 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ex8vX8ChAU92lKxXY07tDCWpX34= 2024-04-09T21:44:53.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ex8vX8ChAU92lKxXY07tDCWpX34=', 'Content-Length': '0'} 2024-04-09T21:44:53.269 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007727efe0e14d3c81-006615b6d5-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.269 DEBUG:boto:path=/xryqqk-6/ 2024-04-09T21:44:53.269 DEBUG:boto:auth_path=/xryqqk-6/ 2024-04-09T21:44:53.269 DEBUG:boto:Method: DELETE 2024-04-09T21:44:53.269 DEBUG:boto:Path: /xryqqk-6/ 2024-04-09T21:44:53.269 DEBUG:boto:Data: 2024-04-09T21:44:53.269 DEBUG:boto:Headers: {} 2024-04-09T21:44:53.269 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:53.269 DEBUG:boto:Port: 8001 2024-04-09T21:44:53.269 DEBUG:boto:Params: {} 2024-04-09T21:44:53.270 DEBUG:boto:Token: None 2024-04-09T21:44:53.270 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:44:53 GMT /xryqqk-6/ 2024-04-09T21:44:53.270 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w3LYN8DEvLTtDiG8P7OmToLGSyk= 2024-04-09T21:44:53.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w3LYN8DEvLTtDiG8P7OmToLGSyk=', 'Content-Length': '0'} 2024-04-09T21:44:53.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd13858f45002b2d-006615b6d5-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:44:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:53.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.322 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:53.370 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:53.370 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:53.400 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:53.412 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:53.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:53.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.414 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699093.297433_67.1", 2024-04-09T21:44:53.414 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:44:53.297433Z" 2024-04-09T21:44:53.414 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.414 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:53.415 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.297433_67.1'} 2024-04-09T21:44:53.415 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:53.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:53.416 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:53.468 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:53.468 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:53.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:53.497+0000 7fe19b982780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:53.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:53.498+0000 7fe19b982780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:53.514 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.515 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.516 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:53.517 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699093.252991_60.1", 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:53.518 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:53.252991Z", 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:53.519 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:53Z" 2024-04-09T21:44:53.520 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:53.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699093.252991_60.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:53.252991Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:53Z" } 2024-04-09T21:44:53.521 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699093.252991_60.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:53.252991Z', 'realm_epoch': 2}}] 2024-04-09T21:44:53.521 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.297433_67.1'} 2024-04-09T21:44:53.521 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.252991_60.1'} 2024-04-09T21:44:53.521 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712699093.297433_67.1 target=1_1712699093.252991_60.1 2024-04-09T21:44:58.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:58.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:58.522 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:58.580 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:58.580 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:58.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:58.613+0000 7fa69ba3c780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:58.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:58.614+0000 7fa69ba3c780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:58.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:58.630 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:58.631 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:58.632 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699093.297433_67.1", 2024-04-09T21:44:58.633 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:53.297433Z", 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:58.634 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:58.635 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:58.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:58.635 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:58Z" 2024-04-09T21:44:58.635 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:58.636 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699093.297433_67.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:53.297433Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:58Z" } 2024-04-09T21:44:58.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699093.297433_67.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:53.297433Z', 'realm_epoch': 2}}] 2024-04-09T21:44:58.636 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.297433_67.1'} 2024-04-09T21:44:58.636 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699093.297433_67.1'} 2024-04-09T21:44:58.636 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:58.637 DEBUG:boto:path=/xryqqk-5/ 2024-04-09T21:44:58.637 DEBUG:boto:auth_path=/xryqqk-5/ 2024-04-09T21:44:58.637 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.637 DEBUG:boto:Path: /xryqqk-5/ 2024-04-09T21:44:58.637 DEBUG:boto:Data: 2024-04-09T21:44:58.637 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.637 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.637 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.637 DEBUG:boto:Params: {} 2024-04-09T21:44:58.638 DEBUG:boto:Token: None 2024-04-09T21:44:58.638 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-5/ 2024-04-09T21:44:58.638 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n5TK5+Lbaue6yNnHz6FSHFkAfGQ= 2024-04-09T21:44:58.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n5TK5+Lbaue6yNnHz6FSHFkAfGQ=', 'Content-Length': '0'} 2024-04-09T21:44:58.640 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000005ce47f64473ea66e-006615b6da-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.640 DEBUG:boto:path=/xryqqk-6/ 2024-04-09T21:44:58.640 DEBUG:boto:auth_path=/xryqqk-6/ 2024-04-09T21:44:58.640 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.640 DEBUG:boto:Path: /xryqqk-6/ 2024-04-09T21:44:58.640 DEBUG:boto:Data: 2024-04-09T21:44:58.640 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.640 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.641 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.641 DEBUG:boto:Params: {} 2024-04-09T21:44:58.641 DEBUG:boto:Token: None 2024-04-09T21:44:58.641 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-6/ 2024-04-09T21:44:58.641 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:77ilSh3wip8xdF6pciOZ4urYUbo= 2024-04-09T21:44:58.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:77ilSh3wip8xdF6pciOZ4urYUbo=', 'Content-Length': '0'} 2024-04-09T21:44:58.642 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000bb80bbf613684646-006615b6da-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.642 DEBUG:boto:path=/xryqqk-5/ 2024-04-09T21:44:58.642 DEBUG:boto:auth_path=/xryqqk-5/ 2024-04-09T21:44:58.642 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.643 DEBUG:boto:Path: /xryqqk-5/ 2024-04-09T21:44:58.643 DEBUG:boto:Data: 2024-04-09T21:44:58.643 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.643 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.643 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.643 DEBUG:boto:Params: {} 2024-04-09T21:44:58.643 DEBUG:boto:Token: None 2024-04-09T21:44:58.643 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-5/ 2024-04-09T21:44:58.643 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n5TK5+Lbaue6yNnHz6FSHFkAfGQ= 2024-04-09T21:44:58.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n5TK5+Lbaue6yNnHz6FSHFkAfGQ=', 'Content-Length': '0'} 2024-04-09T21:44:58.645 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000011809149d986032-006615b6da-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.645 DEBUG:boto:path=/xryqqk-6/ 2024-04-09T21:44:58.645 DEBUG:boto:auth_path=/xryqqk-6/ 2024-04-09T21:44:58.645 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.645 DEBUG:boto:Path: /xryqqk-6/ 2024-04-09T21:44:58.645 DEBUG:boto:Data: 2024-04-09T21:44:58.645 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.645 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.646 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.646 DEBUG:boto:Params: {} 2024-04-09T21:44:58.646 DEBUG:boto:Token: None 2024-04-09T21:44:58.646 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-6/ 2024-04-09T21:44:58.646 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:77ilSh3wip8xdF6pciOZ4urYUbo= 2024-04-09T21:44:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:77ilSh3wip8xdF6pciOZ4urYUbo=', 'Content-Length': '0'} 2024-04-09T21:44:58.647 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx0000089a26167e0cab8c8-006615b6da-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.647 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-04-09T21:44:58.648 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-7 2024-04-09T21:44:58.648 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:44:58.648 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:44:58.648 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.648 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:44:58.648 DEBUG:boto:Data: 2024-04-09T21:44:58.648 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.648 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.649 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.649 DEBUG:boto:Params: {} 2024-04-09T21:44:58.649 DEBUG:boto:Token: None 2024-04-09T21:44:58.649 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/ 2024-04-09T21:44:58.649 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/mYVyAAtxeMJD61V7usrc4YIebE= 2024-04-09T21:44:58.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/mYVyAAtxeMJD61V7usrc4YIebE=', 'Content-Length': '0'} 2024-04-09T21:44:58.667 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000173afd9ce6d4ffdc-006615b6da-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.667 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-8 2024-04-09T21:44:58.667 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:44:58.667 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:44:58.667 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.668 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:44:58.668 DEBUG:boto:Data: 2024-04-09T21:44:58.668 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.668 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.668 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.668 DEBUG:boto:Params: {} 2024-04-09T21:44:58.668 DEBUG:boto:Token: None 2024-04-09T21:44:58.668 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/ 2024-04-09T21:44:58.668 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DeyE/7ux/UMHOdN7gmd2qemw7E4= 2024-04-09T21:44:58.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DeyE/7ux/UMHOdN7gmd2qemw7E4=', 'Content-Length': '0'} 2024-04-09T21:44:58.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000039493d35897ed005-006615b6da-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.693 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:44:58.693 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:44:58.693 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.693 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:44:58.693 DEBUG:boto:Data: 2024-04-09T21:44:58.693 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.693 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.694 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.694 DEBUG:boto:Params: {} 2024-04-09T21:44:58.694 DEBUG:boto:Token: None 2024-04-09T21:44:58.694 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/ 2024-04-09T21:44:58.694 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA= 2024-04-09T21:44:58.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA=', 'Content-Length': '0'} 2024-04-09T21:44:58.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000229651256afa0a77-006615b6da-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.697 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:44:58.697 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:44:58.697 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.697 DEBUG:boto:Path: /xryqqk-7/myobj 2024-04-09T21:44:58.697 DEBUG:boto:Data: 2024-04-09T21:44:58.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.697 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.697 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.697 DEBUG:boto:Params: {} 2024-04-09T21:44:58.697 DEBUG:boto:Token: None 2024-04-09T21:44:58.698 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/myobj 2024-04-09T21:44:58.698 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XTEPv4yGlGwUhlwFs9I5Od02fmc= 2024-04-09T21:44:58.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XTEPv4yGlGwUhlwFs9I5Od02fmc='} 2024-04-09T21:44:58.742 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb688f93adb50a34-006615b6da-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.743 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:44:58.743 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:44:58.743 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.743 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:44:58.743 DEBUG:boto:Data: 2024-04-09T21:44:58.743 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.743 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.743 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.743 DEBUG:boto:Params: {} 2024-04-09T21:44:58.743 DEBUG:boto:Token: None 2024-04-09T21:44:58.744 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/ 2024-04-09T21:44:58.744 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA= 2024-04-09T21:44:58.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA=', 'Content-Length': '0'} 2024-04-09T21:44:58.746 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080067b6153049be6-006615b6da-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.746 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:44:58.746 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:44:58.746 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.746 DEBUG:boto:Path: /xryqqk-7/_myobj 2024-04-09T21:44:58.746 DEBUG:boto:Data: 2024-04-09T21:44:58.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.746 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.746 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.747 DEBUG:boto:Params: {} 2024-04-09T21:44:58.747 DEBUG:boto:Token: None 2024-04-09T21:44:58.747 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/_myobj 2024-04-09T21:44:58.747 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:H0otNMorf0808yxLIuzoReTO91k= 2024-04-09T21:44:58.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:H0otNMorf0808yxLIuzoReTO91k='} 2024-04-09T21:44:58.757 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000837dd10764575370-006615b6da-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.757 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:44:58.757 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:44:58.757 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.757 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:44:58.757 DEBUG:boto:Data: 2024-04-09T21:44:58.757 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.757 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.757 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.757 DEBUG:boto:Params: {} 2024-04-09T21:44:58.758 DEBUG:boto:Token: None 2024-04-09T21:44:58.758 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/ 2024-04-09T21:44:58.758 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA= 2024-04-09T21:44:58.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA=', 'Content-Length': '0'} 2024-04-09T21:44:58.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5acef25625fc6d2-006615b6da-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.760 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:44:58.760 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:44:58.760 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.760 DEBUG:boto:Path: /xryqqk-7/%3A 2024-04-09T21:44:58.760 DEBUG:boto:Data: 2024-04-09T21:44:58.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.760 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.760 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.761 DEBUG:boto:Params: {} 2024-04-09T21:44:58.761 DEBUG:boto:Token: None 2024-04-09T21:44:58.761 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/%3A 2024-04-09T21:44:58.761 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:igamlvKOKuwg28r7aqksVHSXNug= 2024-04-09T21:44:58.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:igamlvKOKuwg28r7aqksVHSXNug='} 2024-04-09T21:44:58.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c06e829a2cf05ba-006615b6da-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.769 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:44:58.769 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:44:58.769 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.769 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:44:58.769 DEBUG:boto:Data: 2024-04-09T21:44:58.769 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.769 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.769 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.769 DEBUG:boto:Params: {} 2024-04-09T21:44:58.769 DEBUG:boto:Token: None 2024-04-09T21:44:58.770 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/ 2024-04-09T21:44:58.770 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA= 2024-04-09T21:44:58.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rTew34xmRxSXep1hZgZgJIFAAPA=', 'Content-Length': '0'} 2024-04-09T21:44:58.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d98c497cac2ba4c5-006615b6da-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.772 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:44:58.772 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:44:58.772 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.772 DEBUG:boto:Path: /xryqqk-7/%26 2024-04-09T21:44:58.772 DEBUG:boto:Data: 2024-04-09T21:44:58.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.772 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:44:58.772 DEBUG:boto:Port: 8000 2024-04-09T21:44:58.772 DEBUG:boto:Params: {} 2024-04-09T21:44:58.773 DEBUG:boto:Token: None 2024-04-09T21:44:58.773 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-7/%26 2024-04-09T21:44:58.773 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ym0Cy5VWL1nQ5X/3rqXzyqH0iNM= 2024-04-09T21:44:58.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ym0Cy5VWL1nQ5X/3rqXzyqH0iNM='} 2024-04-09T21:44:58.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000edf2b5717b054d84-006615b6da-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.815 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:44:58.815 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:44:58.815 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.815 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:44:58.815 DEBUG:boto:Data: 2024-04-09T21:44:58.815 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.815 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.815 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.815 DEBUG:boto:Params: {} 2024-04-09T21:44:58.815 DEBUG:boto:Token: None 2024-04-09T21:44:58.815 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/ 2024-04-09T21:44:58.816 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE= 2024-04-09T21:44:58.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE=', 'Content-Length': '0'} 2024-04-09T21:44:58.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8c4d2bb00bd555b-006615b6da-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.818 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:44:58.818 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:44:58.818 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.818 DEBUG:boto:Path: /xryqqk-8/myobj 2024-04-09T21:44:58.818 DEBUG:boto:Data: 2024-04-09T21:44:58.818 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.818 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.819 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.819 DEBUG:boto:Params: {} 2024-04-09T21:44:58.819 DEBUG:boto:Token: None 2024-04-09T21:44:58.819 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/myobj 2024-04-09T21:44:58.819 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3Nx333Neh8XGrVB+uDjUQawqx0Y= 2024-04-09T21:44:58.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3Nx333Neh8XGrVB+uDjUQawqx0Y='} 2024-04-09T21:44:58.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b1c40798b28fff1c-006615b6da-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.828 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:44:58.828 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:44:58.828 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.828 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:44:58.828 DEBUG:boto:Data: 2024-04-09T21:44:58.828 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.829 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.829 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.829 DEBUG:boto:Params: {} 2024-04-09T21:44:58.829 DEBUG:boto:Token: None 2024-04-09T21:44:58.829 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/ 2024-04-09T21:44:58.829 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE= 2024-04-09T21:44:58.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE=', 'Content-Length': '0'} 2024-04-09T21:44:58.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001cfe60040caa6e31-006615b6da-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.831 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:44:58.832 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:44:58.832 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.832 DEBUG:boto:Path: /xryqqk-8/_myobj 2024-04-09T21:44:58.832 DEBUG:boto:Data: 2024-04-09T21:44:58.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.832 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.832 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.832 DEBUG:boto:Params: {} 2024-04-09T21:44:58.832 DEBUG:boto:Token: None 2024-04-09T21:44:58.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/_myobj 2024-04-09T21:44:58.832 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kq7zZkh3sfu/Cl9SyXwCVLSEGP0= 2024-04-09T21:44:58.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kq7zZkh3sfu/Cl9SyXwCVLSEGP0='} 2024-04-09T21:44:58.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000109a9d536b338751-006615b6da-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.875 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:44:58.875 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:44:58.875 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.875 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:44:58.875 DEBUG:boto:Data: 2024-04-09T21:44:58.875 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.875 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.875 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.875 DEBUG:boto:Params: {} 2024-04-09T21:44:58.876 DEBUG:boto:Token: None 2024-04-09T21:44:58.876 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/ 2024-04-09T21:44:58.876 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE= 2024-04-09T21:44:58.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE=', 'Content-Length': '0'} 2024-04-09T21:44:58.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3ad6c854eacc6a5-006615b6da-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.878 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:44:58.878 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:44:58.878 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.878 DEBUG:boto:Path: /xryqqk-8/%3A 2024-04-09T21:44:58.879 DEBUG:boto:Data: 2024-04-09T21:44:58.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.879 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.879 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.879 DEBUG:boto:Params: {} 2024-04-09T21:44:58.879 DEBUG:boto:Token: None 2024-04-09T21:44:58.879 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/%3A 2024-04-09T21:44:58.879 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9kH1oWwwlsFbaIE163jEaeeWZKI= 2024-04-09T21:44:58.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9kH1oWwwlsFbaIE163jEaeeWZKI='} 2024-04-09T21:44:58.888 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ed24ced6627f52e3-006615b6da-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.888 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:44:58.888 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:44:58.888 DEBUG:boto:Method: HEAD 2024-04-09T21:44:58.888 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:44:58.888 DEBUG:boto:Data: 2024-04-09T21:44:58.888 DEBUG:boto:Headers: {} 2024-04-09T21:44:58.888 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.888 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.888 DEBUG:boto:Params: {} 2024-04-09T21:44:58.888 DEBUG:boto:Token: None 2024-04-09T21:44:58.889 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/ 2024-04-09T21:44:58.889 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE= 2024-04-09T21:44:58.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CpLkmY9NxOYMA80RyXeyANfSHcE=', 'Content-Length': '0'} 2024-04-09T21:44:58.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fac221a553593db0-006615b6da-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.891 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:44:58.891 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:44:58.891 DEBUG:boto:Method: PUT 2024-04-09T21:44:58.891 DEBUG:boto:Path: /xryqqk-8/%26 2024-04-09T21:44:58.891 DEBUG:boto:Data: 2024-04-09T21:44:58.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:44:58.891 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:44:58.891 DEBUG:boto:Port: 8001 2024-04-09T21:44:58.891 DEBUG:boto:Params: {} 2024-04-09T21:44:58.892 DEBUG:boto:Token: None 2024-04-09T21:44:58.892 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:44:58 GMT /xryqqk-8/%26 2024-04-09T21:44:58.892 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TpMhGHEFtJO1XuxXu8cHiUDrgAU= 2024-04-09T21:44:58.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:44:58 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TpMhGHEFtJO1XuxXu8cHiUDrgAU='} 2024-04-09T21:44:58.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006465e9892bf84bd3-006615b6da-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:44:58.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:58.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:58.935 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:58.983 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:58.984 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:59.015 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:44:59.027 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:44:59.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.027 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:59.027 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:44:59.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699098.682169_75.1", 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:44:58.682169Z" 2024-04-09T21:44:59.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:59.029 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:44:59.030 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699098.682169_75.1'} 2024-04-09T21:44:59.030 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:44:59.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:59.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:44:59.030 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:44:59.080 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:59.080 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:59.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:59.107+0000 7f9b28b6f780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:59.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:59.108+0000 7f9b28b6f780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:44:59.121 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:44:59.121 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:44:59.122 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.123 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:59.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:59.125 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699098.682169_75.1", 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:44:58.682169Z", 2024-04-09T21:44:59.126 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:44:59Z" 2024-04-09T21:44:59.127 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:44:59.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699098.682169_75.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:44:58.682169Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:44:59Z" } 2024-04-09T21:44:59.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699098.682169_75.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:44:58.682169Z', 'realm_epoch': 2}}] 2024-04-09T21:44:59.129 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699098.682169_75.1'} 2024-04-09T21:44:59.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '', 3: '1_1712699098.682169_75.1'} 2024-04-09T21:44:59.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:44:59.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:44:59.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:44:59.129 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:44:59.185 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:44:59.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:44:59.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:44:59.217+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:04.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:04.217+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:09.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:09.218+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:14.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:14.218+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:19.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:19.219+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:24.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:24.220+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:29.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:29.220+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:34.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:34.221+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:39.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:39.222+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:44.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:44.222+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:49.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:49.223+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:54.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:54.224+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:45:59.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:45:59.224+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:04.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:04.225+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:09.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:09.226+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:14.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:14.226+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:19.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:19.227+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:24.228+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:29.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:29.228+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:34.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:34.230+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:39.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:39.230+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:44.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:44.230+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:49.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:49.230+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:54.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:54.231+0000 7f4b019b1780 1 waiting to reach incremental sync.. 2024-04-09T21:46:59.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:59.232+0000 7f4b019b1780 1 bucket sync caught up with source: 2024-04-09T21:46:59.235 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.5.6, , 00000000001.5.6, , , , 00000000001.6.6, 00000000001.5.6, , ] 2024-04-09T21:46:59.235 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000001.5.6, , 00000000001.5.6, , , , 00000000001.6.6, 00000000001.5.6, , ] 2024-04-09T21:46:59.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:59.232+0000 7f4b019b1780 0 bucket checkpoint complete 2024-04-09T21:46:59.251 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-7 zones={a2, a1} 2024-04-09T21:46:59.251 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.251 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.251 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.251 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:46:59.251 DEBUG:boto:Data: 2024-04-09T21:46:59.252 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.252 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.252 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.252 DEBUG:boto:Params: {} 2024-04-09T21:46:59.252 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T21:46:59.252 DEBUG:boto:Token: None 2024-04-09T21:46:59.253 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.253 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Xk9GKLBzBVaWLzX+Ol4cMPSc33I= 2024-04-09T21:46:59.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Xk9GKLBzBVaWLzX+Ol4cMPSc33I=', 'Content-Length': '0'} 2024-04-09T21:46:59.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a5fd31688b7d44b-006615b753-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.258 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.258 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.258 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.259 DEBUG:boto:Path: /xryqqk-7/ 2024-04-09T21:46:59.259 DEBUG:boto:Data: 2024-04-09T21:46:59.259 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.259 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.259 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.259 DEBUG:boto:Params: {} 2024-04-09T21:46:59.259 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T21:46:59.259 DEBUG:boto:Token: None 2024-04-09T21:46:59.259 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.259 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Xk9GKLBzBVaWLzX+Ol4cMPSc33I= 2024-04-09T21:46:59.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Xk9GKLBzBVaWLzX+Ol4cMPSc33I=', 'Content-Length': '0'} 2024-04-09T21:46:59.264 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000752aca914408c2fc-006615b753-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:46:59.264 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.265 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.265 DEBUG:boto:path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.265 DEBUG:boto:auth_path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.265 DEBUG:boto:Method: GET 2024-04-09T21:46:59.265 DEBUG:boto:Path: /xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.265 DEBUG:boto:Data: 2024-04-09T21:46:59.265 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.265 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.265 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.265 DEBUG:boto:Params: {} 2024-04-09T21:46:59.266 DEBUG:boto:Token: None 2024-04-09T21:46:59.266 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/?versions 2024-04-09T21:46:59.266 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI= 2024-04-09T21:46:59.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI=', 'Content-Length': '0'} 2024-04-09T21:46:59.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f21bb19f27e97f6f-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.268 DEBUG:boto:b'xryqqk-7999false&nulltrue2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-09T21:46:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-09T21:46:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-09T21:46:59.271 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:46:59.271 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:46:59.271 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.271 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.272 DEBUG:boto:path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.272 DEBUG:boto:auth_path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.272 DEBUG:boto:Method: GET 2024-04-09T21:46:59.272 DEBUG:boto:Path: /xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.272 DEBUG:boto:Data: 2024-04-09T21:46:59.272 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.272 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.272 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.272 DEBUG:boto:Params: {} 2024-04-09T21:46:59.272 DEBUG:boto:Token: None 2024-04-09T21:46:59.272 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/?versions 2024-04-09T21:46:59.272 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI= 2024-04-09T21:46:59.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI=', 'Content-Length': '0'} 2024-04-09T21:46:59.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2842bce9ca30fac-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.275 DEBUG:boto:b'xryqqk-7999false&nulltrue2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.275 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-09T21:46:59.276 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-09T21:46:59.276 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-09T21:46:59.276 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:46:59.276 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.276 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.276 DEBUG:boto:path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.276 DEBUG:boto:auth_path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.276 DEBUG:boto:Method: GET 2024-04-09T21:46:59.276 DEBUG:boto:Path: /xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.276 DEBUG:boto:Data: 2024-04-09T21:46:59.276 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.276 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.276 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.277 DEBUG:boto:Params: {} 2024-04-09T21:46:59.277 DEBUG:boto:Token: None 2024-04-09T21:46:59.277 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/?versions 2024-04-09T21:46:59.277 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI= 2024-04-09T21:46:59.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI=', 'Content-Length': '0'} 2024-04-09T21:46:59.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009931e092ce588c2c-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.279 DEBUG:boto:b'xryqqk-7999false&nulltrue2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.280 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.280 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.280 DEBUG:boto:path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.280 DEBUG:boto:auth_path=/xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.280 DEBUG:boto:Method: GET 2024-04-09T21:46:59.280 DEBUG:boto:Path: /xryqqk-7/?versions&max-keys=999 2024-04-09T21:46:59.280 DEBUG:boto:Data: 2024-04-09T21:46:59.280 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.280 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.280 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.280 DEBUG:boto:Params: {} 2024-04-09T21:46:59.280 DEBUG:boto:Token: None 2024-04-09T21:46:59.281 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/?versions 2024-04-09T21:46:59.281 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI= 2024-04-09T21:46:59.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+8kNDQpy6APsqvpnXditpBQoaoI=', 'Content-Length': '0'} 2024-04-09T21:46:59.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000991d574146242608-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.283 DEBUG:boto:b'xryqqk-7999false&nulltrue2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-09T21:46:59.284 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.284 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.284 DEBUG:boto:path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.284 DEBUG:boto:auth_path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.284 DEBUG:boto:Method: GET 2024-04-09T21:46:59.284 DEBUG:boto:Path: /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.284 DEBUG:boto:Data: 2024-04-09T21:46:59.284 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.284 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.284 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.284 DEBUG:boto:Params: {} 2024-04-09T21:46:59.285 DEBUG:boto:Token: None 2024-04-09T21:46:59.285 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.285 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA= 2024-04-09T21:46:59.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA=', 'Content-Length': '0'} 2024-04-09T21:46:59.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c7d8558f78782c9-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.287 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.287 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.287 DEBUG:boto:path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.287 DEBUG:boto:auth_path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.287 DEBUG:boto:Method: GET 2024-04-09T21:46:59.287 DEBUG:boto:Path: /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.287 DEBUG:boto:Data: 2024-04-09T21:46:59.288 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.288 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.288 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.288 DEBUG:boto:Params: {} 2024-04-09T21:46:59.288 DEBUG:boto:Token: None 2024-04-09T21:46:59.288 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.288 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA= 2024-04-09T21:46:59.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA=', 'Content-Length': '0'} 2024-04-09T21:46:59.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed55dc0e8a70cd45-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.290 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.290 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.291 DEBUG:boto:path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.291 DEBUG:boto:auth_path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.291 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.291 DEBUG:boto:Path: /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.291 DEBUG:boto:Data: 2024-04-09T21:46:59.291 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.291 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.291 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.291 DEBUG:boto:Params: {} 2024-04-09T21:46:59.291 DEBUG:boto:Token: None 2024-04-09T21:46:59.292 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.292 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yyazqmHSqDvKbi3JIfFHQvkV7B0= 2024-04-09T21:46:59.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yyazqmHSqDvKbi3JIfFHQvkV7B0=', 'Content-Length': '0'} 2024-04-09T21:46:59.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac00bb3918070718-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.294 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.294 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.294 DEBUG:boto:path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.294 DEBUG:boto:auth_path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.294 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.294 DEBUG:boto:Path: /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.294 DEBUG:boto:Data: 2024-04-09T21:46:59.294 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.294 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.294 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.294 DEBUG:boto:Params: {} 2024-04-09T21:46:59.295 DEBUG:boto:Token: None 2024-04-09T21:46:59.295 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.295 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yyazqmHSqDvKbi3JIfFHQvkV7B0= 2024-04-09T21:46:59.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yyazqmHSqDvKbi3JIfFHQvkV7B0=', 'Content-Length': '0'} 2024-04-09T21:46:59.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048ee87cba471b4ba-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-09T21:46:59.297 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.297 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.297 DEBUG:boto:path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.297 DEBUG:boto:auth_path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.297 DEBUG:boto:Method: GET 2024-04-09T21:46:59.297 DEBUG:boto:Path: /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.297 DEBUG:boto:Data: 2024-04-09T21:46:59.297 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.298 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.298 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.298 DEBUG:boto:Params: {} 2024-04-09T21:46:59.298 DEBUG:boto:Token: None 2024-04-09T21:46:59.298 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.298 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA= 2024-04-09T21:46:59.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA=', 'Content-Length': '0'} 2024-04-09T21:46:59.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027e7319ed7391f8a-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.300 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.300 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.300 DEBUG:boto:path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.301 DEBUG:boto:auth_path=/xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.301 DEBUG:boto:Method: GET 2024-04-09T21:46:59.301 DEBUG:boto:Path: /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.301 DEBUG:boto:Data: 2024-04-09T21:46:59.301 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.301 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.301 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.301 DEBUG:boto:Params: {} 2024-04-09T21:46:59.301 DEBUG:boto:Token: None 2024-04-09T21:46:59.301 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26?versionId=null 2024-04-09T21:46:59.301 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA= 2024-04-09T21:46:59.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:maF42Zi+KQL6iZkwICZusAoYpCA=', 'Content-Length': '0'} 2024-04-09T21:46:59.303 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f824f3b4a37c243-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.304 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.304 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.304 DEBUG:boto:path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.304 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.304 DEBUG:boto:Method: GET 2024-04-09T21:46:59.304 DEBUG:boto:Path: /xryqqk-7/?prefix=%26 2024-04-09T21:46:59.304 DEBUG:boto:Data: 2024-04-09T21:46:59.304 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.304 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.304 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.304 DEBUG:boto:Params: {} 2024-04-09T21:46:59.305 DEBUG:boto:Token: None 2024-04-09T21:46:59.305 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.305 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4a7ecc406b67955-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.307 DEBUG:boto:b'xryqqk-7&1000false&2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.307 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.307 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.308 DEBUG:boto:path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.308 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.308 DEBUG:boto:Method: GET 2024-04-09T21:46:59.308 DEBUG:boto:Path: /xryqqk-7/?prefix=%26 2024-04-09T21:46:59.308 DEBUG:boto:Data: 2024-04-09T21:46:59.308 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.308 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.308 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.308 DEBUG:boto:Params: {} 2024-04-09T21:46:59.308 DEBUG:boto:Token: None 2024-04-09T21:46:59.308 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.309 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b14c286738b325d2-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.354 DEBUG:boto:b'xryqqk-7&1000false&2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.355 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.355 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.355 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.355 DEBUG:boto:Path: /xryqqk-7/%26 2024-04-09T21:46:59.355 DEBUG:boto:Data: 2024-04-09T21:46:59.355 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.356 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.356 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.356 DEBUG:boto:Params: {} 2024-04-09T21:46:59.356 DEBUG:boto:Token: None 2024-04-09T21:46:59.356 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26 2024-04-09T21:46:59.356 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LvgrjsO8DYQ5VGZzANqulPiozsw= 2024-04-09T21:46:59.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LvgrjsO8DYQ5VGZzANqulPiozsw=', 'Content-Length': '0'} 2024-04-09T21:46:59.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b9e38b852c422fe-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.358 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.358 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.358 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.358 DEBUG:boto:Path: /xryqqk-7/%26 2024-04-09T21:46:59.359 DEBUG:boto:Data: 2024-04-09T21:46:59.359 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.359 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.359 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.359 DEBUG:boto:Params: {} 2024-04-09T21:46:59.359 DEBUG:boto:Token: None 2024-04-09T21:46:59.359 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26 2024-04-09T21:46:59.359 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LvgrjsO8DYQ5VGZzANqulPiozsw= 2024-04-09T21:46:59.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LvgrjsO8DYQ5VGZzANqulPiozsw=', 'Content-Length': '0'} 2024-04-09T21:46:59.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005393e295d5b39daf-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-09T21:46:59.361 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.361 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.361 DEBUG:boto:Method: GET 2024-04-09T21:46:59.361 DEBUG:boto:Path: /xryqqk-7/%26 2024-04-09T21:46:59.361 DEBUG:boto:Data: 2024-04-09T21:46:59.362 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.362 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.362 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.362 DEBUG:boto:Params: {} 2024-04-09T21:46:59.362 DEBUG:boto:Token: None 2024-04-09T21:46:59.362 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26 2024-04-09T21:46:59.362 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oaKzDYB5PmxnBngv//AzsenhZPY= 2024-04-09T21:46:59.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oaKzDYB5PmxnBngv//AzsenhZPY=', 'Content-Length': '0'} 2024-04-09T21:46:59.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000851884e1851821bf-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.364 DEBUG:boto:path=/xryqqk-7/%26 2024-04-09T21:46:59.364 DEBUG:boto:auth_path=/xryqqk-7/%26 2024-04-09T21:46:59.364 DEBUG:boto:Method: GET 2024-04-09T21:46:59.364 DEBUG:boto:Path: /xryqqk-7/%26 2024-04-09T21:46:59.364 DEBUG:boto:Data: 2024-04-09T21:46:59.364 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.365 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.365 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.365 DEBUG:boto:Params: {} 2024-04-09T21:46:59.365 DEBUG:boto:Token: None 2024-04-09T21:46:59.365 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%26 2024-04-09T21:46:59.365 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oaKzDYB5PmxnBngv//AzsenhZPY= 2024-04-09T21:46:59.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oaKzDYB5PmxnBngv//AzsenhZPY=', 'Content-Length': '0'} 2024-04-09T21:46:59.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e395fe0b8a5f0d67-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.367 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.367 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.367 DEBUG:boto:path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.368 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.368 DEBUG:boto:Method: GET 2024-04-09T21:46:59.368 DEBUG:boto:Path: /xryqqk-7/?prefix=%26 2024-04-09T21:46:59.368 DEBUG:boto:Data: 2024-04-09T21:46:59.368 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.368 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.368 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.368 DEBUG:boto:Params: {} 2024-04-09T21:46:59.368 DEBUG:boto:Token: None 2024-04-09T21:46:59.368 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.368 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030550cfc7e3bab5c-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.371 DEBUG:boto:b'xryqqk-7&1000false&2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.371 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.371 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.371 DEBUG:boto:path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.371 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%26 2024-04-09T21:46:59.371 DEBUG:boto:Method: GET 2024-04-09T21:46:59.371 DEBUG:boto:Path: /xryqqk-7/?prefix=%26 2024-04-09T21:46:59.371 DEBUG:boto:Data: 2024-04-09T21:46:59.372 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.372 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.372 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.372 DEBUG:boto:Params: {} 2024-04-09T21:46:59.372 DEBUG:boto:Token: None 2024-04-09T21:46:59.372 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.372 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab171a0a47036192-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.375 DEBUG:boto:b'xryqqk-7&1000false&2024-04-09T21:44:58.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-09T21:46:59.375 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.375 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.375 DEBUG:boto:path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.375 DEBUG:boto:auth_path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.375 DEBUG:boto:Method: GET 2024-04-09T21:46:59.375 DEBUG:boto:Path: /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.375 DEBUG:boto:Data: 2024-04-09T21:46:59.376 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.376 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.376 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.376 DEBUG:boto:Params: {} 2024-04-09T21:46:59.376 DEBUG:boto:Token: None 2024-04-09T21:46:59.376 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.376 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY= 2024-04-09T21:46:59.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY=', 'Content-Length': '0'} 2024-04-09T21:46:59.378 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdd0579c7b123f38-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.422 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.422 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.423 DEBUG:boto:path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.423 DEBUG:boto:auth_path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.423 DEBUG:boto:Method: GET 2024-04-09T21:46:59.423 DEBUG:boto:Path: /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.423 DEBUG:boto:Data: 2024-04-09T21:46:59.423 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.423 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.423 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.423 DEBUG:boto:Params: {} 2024-04-09T21:46:59.423 DEBUG:boto:Token: None 2024-04-09T21:46:59.423 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.424 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY= 2024-04-09T21:46:59.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY=', 'Content-Length': '0'} 2024-04-09T21:46:59.425 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ce135c597db867-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.426 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.426 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.426 DEBUG:boto:path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.426 DEBUG:boto:auth_path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.426 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.426 DEBUG:boto:Path: /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.426 DEBUG:boto:Data: 2024-04-09T21:46:59.426 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.426 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.426 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.427 DEBUG:boto:Params: {} 2024-04-09T21:46:59.427 DEBUG:boto:Token: None 2024-04-09T21:46:59.427 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.427 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t3ghPzBKpi/kpO+j91EUxA95mkk= 2024-04-09T21:46:59.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t3ghPzBKpi/kpO+j91EUxA95mkk=', 'Content-Length': '0'} 2024-04-09T21:46:59.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a1e38f403e1aef8-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.429 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.429 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.429 DEBUG:boto:path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.429 DEBUG:boto:auth_path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.429 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.429 DEBUG:boto:Path: /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.430 DEBUG:boto:Data: 2024-04-09T21:46:59.430 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.430 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.430 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.430 DEBUG:boto:Params: {} 2024-04-09T21:46:59.430 DEBUG:boto:Token: None 2024-04-09T21:46:59.430 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.430 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t3ghPzBKpi/kpO+j91EUxA95mkk= 2024-04-09T21:46:59.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t3ghPzBKpi/kpO+j91EUxA95mkk=', 'Content-Length': '0'} 2024-04-09T21:46:59.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015e57865c0f15798-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-09T21:46:59.433 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.433 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.433 DEBUG:boto:path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.433 DEBUG:boto:auth_path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.433 DEBUG:boto:Method: GET 2024-04-09T21:46:59.433 DEBUG:boto:Path: /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.433 DEBUG:boto:Data: 2024-04-09T21:46:59.433 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.433 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.433 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.433 DEBUG:boto:Params: {} 2024-04-09T21:46:59.434 DEBUG:boto:Token: None 2024-04-09T21:46:59.434 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.434 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY= 2024-04-09T21:46:59.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY=', 'Content-Length': '0'} 2024-04-09T21:46:59.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006193544098b50879-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.436 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.436 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.436 DEBUG:boto:path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.436 DEBUG:boto:auth_path=/xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.436 DEBUG:boto:Method: GET 2024-04-09T21:46:59.436 DEBUG:boto:Path: /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.437 DEBUG:boto:Data: 2024-04-09T21:46:59.437 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.437 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.437 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.437 DEBUG:boto:Params: {} 2024-04-09T21:46:59.437 DEBUG:boto:Token: None 2024-04-09T21:46:59.437 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A?versionId=null 2024-04-09T21:46:59.437 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY= 2024-04-09T21:46:59.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i79BXjL0yf/BXzmT+ODDAQTrHFY=', 'Content-Length': '0'} 2024-04-09T21:46:59.439 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000944426edbba16e6c-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.439 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.439 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.439 DEBUG:boto:path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.439 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.440 DEBUG:boto:Method: GET 2024-04-09T21:46:59.440 DEBUG:boto:Path: /xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.440 DEBUG:boto:Data: 2024-04-09T21:46:59.440 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.440 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.440 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.440 DEBUG:boto:Params: {} 2024-04-09T21:46:59.440 DEBUG:boto:Token: None 2024-04-09T21:46:59.440 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.440 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090eb4a154aeecae8-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.443 DEBUG:boto:b'xryqqk-7:1000false:2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.443 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.443 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.443 DEBUG:boto:path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.443 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.443 DEBUG:boto:Method: GET 2024-04-09T21:46:59.443 DEBUG:boto:Path: /xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.444 DEBUG:boto:Data: 2024-04-09T21:46:59.444 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.444 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.444 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.444 DEBUG:boto:Params: {} 2024-04-09T21:46:59.444 DEBUG:boto:Token: None 2024-04-09T21:46:59.444 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.444 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbdd0490673d17c0-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.490 DEBUG:boto:b'xryqqk-7:1000false:2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.491 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.491 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.491 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.491 DEBUG:boto:Path: /xryqqk-7/%3A 2024-04-09T21:46:59.491 DEBUG:boto:Data: 2024-04-09T21:46:59.491 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.491 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.491 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.491 DEBUG:boto:Params: {} 2024-04-09T21:46:59.491 DEBUG:boto:Token: None 2024-04-09T21:46:59.492 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A 2024-04-09T21:46:59.492 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Z03Nw7PUZvwTLx9v5J6vWuJV4XY= 2024-04-09T21:46:59.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Z03Nw7PUZvwTLx9v5J6vWuJV4XY=', 'Content-Length': '0'} 2024-04-09T21:46:59.494 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045751a88006e9127-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.494 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.494 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.494 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.494 DEBUG:boto:Path: /xryqqk-7/%3A 2024-04-09T21:46:59.494 DEBUG:boto:Data: 2024-04-09T21:46:59.494 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.494 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.495 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.495 DEBUG:boto:Params: {} 2024-04-09T21:46:59.495 DEBUG:boto:Token: None 2024-04-09T21:46:59.495 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A 2024-04-09T21:46:59.495 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Z03Nw7PUZvwTLx9v5J6vWuJV4XY= 2024-04-09T21:46:59.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Z03Nw7PUZvwTLx9v5J6vWuJV4XY=', 'Content-Length': '0'} 2024-04-09T21:46:59.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6de8d199e8b8e39-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-09T21:46:59.497 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.497 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.497 DEBUG:boto:Method: GET 2024-04-09T21:46:59.497 DEBUG:boto:Path: /xryqqk-7/%3A 2024-04-09T21:46:59.497 DEBUG:boto:Data: 2024-04-09T21:46:59.497 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.497 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.498 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.498 DEBUG:boto:Params: {} 2024-04-09T21:46:59.498 DEBUG:boto:Token: None 2024-04-09T21:46:59.498 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A 2024-04-09T21:46:59.498 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2+csSFyyfmA4mss6ebM6WGZPEc8= 2024-04-09T21:46:59.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2+csSFyyfmA4mss6ebM6WGZPEc8=', 'Content-Length': '0'} 2024-04-09T21:46:59.500 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d933bf3843d29f1d-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.500 DEBUG:boto:path=/xryqqk-7/%3A 2024-04-09T21:46:59.500 DEBUG:boto:auth_path=/xryqqk-7/%3A 2024-04-09T21:46:59.500 DEBUG:boto:Method: GET 2024-04-09T21:46:59.500 DEBUG:boto:Path: /xryqqk-7/%3A 2024-04-09T21:46:59.501 DEBUG:boto:Data: 2024-04-09T21:46:59.501 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.501 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.501 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.501 DEBUG:boto:Params: {} 2024-04-09T21:46:59.501 DEBUG:boto:Token: None 2024-04-09T21:46:59.501 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/%3A 2024-04-09T21:46:59.501 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2+csSFyyfmA4mss6ebM6WGZPEc8= 2024-04-09T21:46:59.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2+csSFyyfmA4mss6ebM6WGZPEc8=', 'Content-Length': '0'} 2024-04-09T21:46:59.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d42ecba647c07bc3-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.503 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.503 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.503 DEBUG:boto:path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.503 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.504 DEBUG:boto:Method: GET 2024-04-09T21:46:59.504 DEBUG:boto:Path: /xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.504 DEBUG:boto:Data: 2024-04-09T21:46:59.504 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.504 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.504 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.504 DEBUG:boto:Params: {} 2024-04-09T21:46:59.504 DEBUG:boto:Token: None 2024-04-09T21:46:59.504 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.504 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8add9b8b8ad130a-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.507 DEBUG:boto:b'xryqqk-7:1000false:2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.507 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.507 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.507 DEBUG:boto:path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.507 DEBUG:boto:auth_path=/xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.507 DEBUG:boto:Method: GET 2024-04-09T21:46:59.508 DEBUG:boto:Path: /xryqqk-7/?prefix=%3A 2024-04-09T21:46:59.508 DEBUG:boto:Data: 2024-04-09T21:46:59.508 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.508 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.508 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.508 DEBUG:boto:Params: {} 2024-04-09T21:46:59.508 DEBUG:boto:Token: None 2024-04-09T21:46:59.508 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.508 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b5ce1745037d4d5-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.511 DEBUG:boto:b'xryqqk-7:1000false:2024-04-09T21:44:58.762Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-09T21:46:59.511 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.511 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.511 DEBUG:boto:path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.512 DEBUG:boto:auth_path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.512 DEBUG:boto:Method: GET 2024-04-09T21:46:59.512 DEBUG:boto:Path: /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.512 DEBUG:boto:Data: 2024-04-09T21:46:59.512 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.512 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.512 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.512 DEBUG:boto:Params: {} 2024-04-09T21:46:59.512 DEBUG:boto:Token: None 2024-04-09T21:46:59.512 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.512 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A= 2024-04-09T21:46:59.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A=', 'Content-Length': '0'} 2024-04-09T21:46:59.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e166fefc5c44622e-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.515 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.515 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.515 DEBUG:boto:path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.515 DEBUG:boto:auth_path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.515 DEBUG:boto:Method: GET 2024-04-09T21:46:59.515 DEBUG:boto:Path: /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.515 DEBUG:boto:Data: 2024-04-09T21:46:59.515 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.515 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.515 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.516 DEBUG:boto:Params: {} 2024-04-09T21:46:59.516 DEBUG:boto:Token: None 2024-04-09T21:46:59.516 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.516 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A= 2024-04-09T21:46:59.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A=', 'Content-Length': '0'} 2024-04-09T21:46:59.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7073e881d1b7ad5-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.558 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.558 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.558 DEBUG:boto:path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.559 DEBUG:boto:auth_path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.559 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.559 DEBUG:boto:Path: /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.559 DEBUG:boto:Data: 2024-04-09T21:46:59.559 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.559 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.559 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.559 DEBUG:boto:Params: {} 2024-04-09T21:46:59.559 DEBUG:boto:Token: None 2024-04-09T21:46:59.559 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.559 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:isdbCLwTiP4s31Cb+xpyDwDp6XQ= 2024-04-09T21:46:59.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:isdbCLwTiP4s31Cb+xpyDwDp6XQ=', 'Content-Length': '0'} 2024-04-09T21:46:59.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022d8398537f0567b-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.562 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.562 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.562 DEBUG:boto:path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.562 DEBUG:boto:auth_path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.562 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.562 DEBUG:boto:Path: /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.562 DEBUG:boto:Data: 2024-04-09T21:46:59.562 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.562 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.562 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.562 DEBUG:boto:Params: {} 2024-04-09T21:46:59.563 DEBUG:boto:Token: None 2024-04-09T21:46:59.563 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.563 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:isdbCLwTiP4s31Cb+xpyDwDp6XQ= 2024-04-09T21:46:59.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:isdbCLwTiP4s31Cb+xpyDwDp6XQ=', 'Content-Length': '0'} 2024-04-09T21:46:59.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adf65b8a5a11152a-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-09T21:46:59.565 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.565 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.565 DEBUG:boto:path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.565 DEBUG:boto:auth_path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.565 DEBUG:boto:Method: GET 2024-04-09T21:46:59.565 DEBUG:boto:Path: /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.565 DEBUG:boto:Data: 2024-04-09T21:46:59.565 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.565 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.566 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.566 DEBUG:boto:Params: {} 2024-04-09T21:46:59.566 DEBUG:boto:Token: None 2024-04-09T21:46:59.566 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.566 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A= 2024-04-09T21:46:59.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A=', 'Content-Length': '0'} 2024-04-09T21:46:59.568 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d30c0613a899251a-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.568 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.568 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.568 DEBUG:boto:path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.568 DEBUG:boto:auth_path=/xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.568 DEBUG:boto:Method: GET 2024-04-09T21:46:59.568 DEBUG:boto:Path: /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.569 DEBUG:boto:Data: 2024-04-09T21:46:59.569 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.569 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.569 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.569 DEBUG:boto:Params: {} 2024-04-09T21:46:59.569 DEBUG:boto:Token: None 2024-04-09T21:46:59.569 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj?versionId=null 2024-04-09T21:46:59.569 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A= 2024-04-09T21:46:59.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/GNGnN8MQC0JrqGvplJUfyVVr1A=', 'Content-Length': '0'} 2024-04-09T21:46:59.571 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014e8b0201d63a058-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.571 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.571 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.571 DEBUG:boto:path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.572 DEBUG:boto:auth_path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.572 DEBUG:boto:Method: GET 2024-04-09T21:46:59.572 DEBUG:boto:Path: /xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.572 DEBUG:boto:Data: 2024-04-09T21:46:59.572 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.572 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.572 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.572 DEBUG:boto:Params: {} 2024-04-09T21:46:59.572 DEBUG:boto:Token: None 2024-04-09T21:46:59.572 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.572 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd779815976859f5-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.575 DEBUG:boto:b'xryqqk-7_myobj1000false_myobj2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.575 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.575 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.575 DEBUG:boto:path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.575 DEBUG:boto:auth_path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.576 DEBUG:boto:Method: GET 2024-04-09T21:46:59.576 DEBUG:boto:Path: /xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.576 DEBUG:boto:Data: 2024-04-09T21:46:59.576 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.576 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.576 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.576 DEBUG:boto:Params: {} 2024-04-09T21:46:59.576 DEBUG:boto:Token: None 2024-04-09T21:46:59.576 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.576 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000834c5a3b8640d9eb-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.579 DEBUG:boto:b'xryqqk-7_myobj1000false_myobj2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.579 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.579 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.579 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.579 DEBUG:boto:Path: /xryqqk-7/_myobj 2024-04-09T21:46:59.579 DEBUG:boto:Data: 2024-04-09T21:46:59.579 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.579 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.580 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.580 DEBUG:boto:Params: {} 2024-04-09T21:46:59.580 DEBUG:boto:Token: None 2024-04-09T21:46:59.580 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj 2024-04-09T21:46:59.580 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3IktwBwahlP+yCFRcZMjlv4kwtQ= 2024-04-09T21:46:59.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3IktwBwahlP+yCFRcZMjlv4kwtQ=', 'Content-Length': '0'} 2024-04-09T21:46:59.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077f88e065e8824ae-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.582 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.582 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.582 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.582 DEBUG:boto:Path: /xryqqk-7/_myobj 2024-04-09T21:46:59.582 DEBUG:boto:Data: 2024-04-09T21:46:59.582 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.582 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.583 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.583 DEBUG:boto:Params: {} 2024-04-09T21:46:59.583 DEBUG:boto:Token: None 2024-04-09T21:46:59.583 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj 2024-04-09T21:46:59.583 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3IktwBwahlP+yCFRcZMjlv4kwtQ= 2024-04-09T21:46:59.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3IktwBwahlP+yCFRcZMjlv4kwtQ=', 'Content-Length': '0'} 2024-04-09T21:46:59.585 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091a38d3dca8feeda-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-09T21:46:59.585 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.585 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.585 DEBUG:boto:Method: GET 2024-04-09T21:46:59.585 DEBUG:boto:Path: /xryqqk-7/_myobj 2024-04-09T21:46:59.585 DEBUG:boto:Data: 2024-04-09T21:46:59.585 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.586 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.586 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.586 DEBUG:boto:Params: {} 2024-04-09T21:46:59.586 DEBUG:boto:Token: None 2024-04-09T21:46:59.586 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj 2024-04-09T21:46:59.586 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mx1jQl+klVcdNSHPneN8gb05We4= 2024-04-09T21:46:59.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mx1jQl+klVcdNSHPneN8gb05We4=', 'Content-Length': '0'} 2024-04-09T21:46:59.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072988be62fb0e045-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.630 DEBUG:boto:path=/xryqqk-7/_myobj 2024-04-09T21:46:59.630 DEBUG:boto:auth_path=/xryqqk-7/_myobj 2024-04-09T21:46:59.631 DEBUG:boto:Method: GET 2024-04-09T21:46:59.631 DEBUG:boto:Path: /xryqqk-7/_myobj 2024-04-09T21:46:59.631 DEBUG:boto:Data: 2024-04-09T21:46:59.631 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.631 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.631 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.631 DEBUG:boto:Params: {} 2024-04-09T21:46:59.631 DEBUG:boto:Token: None 2024-04-09T21:46:59.631 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/_myobj 2024-04-09T21:46:59.631 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mx1jQl+klVcdNSHPneN8gb05We4= 2024-04-09T21:46:59.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mx1jQl+klVcdNSHPneN8gb05We4=', 'Content-Length': '0'} 2024-04-09T21:46:59.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7b19c92dc79f9a3-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.633 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.633 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.633 DEBUG:boto:path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.634 DEBUG:boto:auth_path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.634 DEBUG:boto:Method: GET 2024-04-09T21:46:59.634 DEBUG:boto:Path: /xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.634 DEBUG:boto:Data: 2024-04-09T21:46:59.634 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.634 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.634 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.634 DEBUG:boto:Params: {} 2024-04-09T21:46:59.634 DEBUG:boto:Token: None 2024-04-09T21:46:59.634 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.635 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea55c94122cc324a-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.637 DEBUG:boto:b'xryqqk-7_myobj1000false_myobj2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.637 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.637 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.637 DEBUG:boto:path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.637 DEBUG:boto:auth_path=/xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.637 DEBUG:boto:Method: GET 2024-04-09T21:46:59.638 DEBUG:boto:Path: /xryqqk-7/?prefix=_myobj 2024-04-09T21:46:59.638 DEBUG:boto:Data: 2024-04-09T21:46:59.638 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.638 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.638 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.638 DEBUG:boto:Params: {} 2024-04-09T21:46:59.638 DEBUG:boto:Token: None 2024-04-09T21:46:59.638 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.638 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091d2a001eec5a7b2-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.641 DEBUG:boto:b'xryqqk-7_myobj1000false_myobj2024-04-09T21:44:58.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:46:59.641 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.641 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.641 DEBUG:boto:path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.641 DEBUG:boto:auth_path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.641 DEBUG:boto:Method: GET 2024-04-09T21:46:59.641 DEBUG:boto:Path: /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.641 DEBUG:boto:Data: 2024-04-09T21:46:59.642 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.642 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.642 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.642 DEBUG:boto:Params: {} 2024-04-09T21:46:59.642 DEBUG:boto:Token: None 2024-04-09T21:46:59.642 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.642 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o= 2024-04-09T21:46:59.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o=', 'Content-Length': '0'} 2024-04-09T21:46:59.644 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008d8698b8c910eb8a-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.644 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.645 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.645 DEBUG:boto:path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.645 DEBUG:boto:auth_path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.645 DEBUG:boto:Method: GET 2024-04-09T21:46:59.645 DEBUG:boto:Path: /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.645 DEBUG:boto:Data: 2024-04-09T21:46:59.645 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.645 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.645 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.645 DEBUG:boto:Params: {} 2024-04-09T21:46:59.646 DEBUG:boto:Token: None 2024-04-09T21:46:59.646 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.646 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o= 2024-04-09T21:46:59.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o=', 'Content-Length': '0'} 2024-04-09T21:46:59.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078c6018bbc08da98-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.648 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.648 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.648 DEBUG:boto:path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.648 DEBUG:boto:auth_path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.648 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.648 DEBUG:boto:Path: /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.648 DEBUG:boto:Data: 2024-04-09T21:46:59.648 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.648 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.648 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.649 DEBUG:boto:Params: {} 2024-04-09T21:46:59.649 DEBUG:boto:Token: None 2024-04-09T21:46:59.649 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.649 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HMWjCue03j3mRQDOJLQBa4mMXTw= 2024-04-09T21:46:59.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HMWjCue03j3mRQDOJLQBa4mMXTw=', 'Content-Length': '0'} 2024-04-09T21:46:59.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045ad612ddcd9adb7-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.651 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.651 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.651 DEBUG:boto:path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.651 DEBUG:boto:auth_path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.651 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.651 DEBUG:boto:Path: /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.651 DEBUG:boto:Data: 2024-04-09T21:46:59.652 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.652 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.652 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.652 DEBUG:boto:Params: {} 2024-04-09T21:46:59.652 DEBUG:boto:Token: None 2024-04-09T21:46:59.652 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.652 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HMWjCue03j3mRQDOJLQBa4mMXTw= 2024-04-09T21:46:59.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HMWjCue03j3mRQDOJLQBa4mMXTw=', 'Content-Length': '0'} 2024-04-09T21:46:59.654 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbc1180e35237f88-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:46:59.654 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.654 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.654 DEBUG:boto:path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.655 DEBUG:boto:auth_path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.655 DEBUG:boto:Method: GET 2024-04-09T21:46:59.655 DEBUG:boto:Path: /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.655 DEBUG:boto:Data: 2024-04-09T21:46:59.655 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.655 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.655 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.655 DEBUG:boto:Params: {} 2024-04-09T21:46:59.655 DEBUG:boto:Token: None 2024-04-09T21:46:59.655 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o= 2024-04-09T21:46:59.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o=', 'Content-Length': '0'} 2024-04-09T21:46:59.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e17ed60716332c75-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.658 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.658 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.658 DEBUG:boto:path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.658 DEBUG:boto:auth_path=/xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.658 DEBUG:boto:Method: GET 2024-04-09T21:46:59.658 DEBUG:boto:Path: /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.658 DEBUG:boto:Data: 2024-04-09T21:46:59.658 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.658 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.658 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.658 DEBUG:boto:Params: {} 2024-04-09T21:46:59.659 DEBUG:boto:Token: None 2024-04-09T21:46:59.659 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj?versionId=null 2024-04-09T21:46:59.659 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o= 2024-04-09T21:46:59.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:abi39sj340tdT+ydueSGP0cdr4o=', 'Content-Length': '0'} 2024-04-09T21:46:59.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096c472d21b8d0ce4-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.661 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.661 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.661 DEBUG:boto:path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.661 DEBUG:boto:auth_path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.661 DEBUG:boto:Method: GET 2024-04-09T21:46:59.661 DEBUG:boto:Path: /xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.661 DEBUG:boto:Data: 2024-04-09T21:46:59.661 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.661 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.661 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.661 DEBUG:boto:Params: {} 2024-04-09T21:46:59.662 DEBUG:boto:Token: None 2024-04-09T21:46:59.662 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.662 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d6fe6c6857211c9-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.664 DEBUG:boto:b'xryqqk-7myobj1000falsemyobj2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.665 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.665 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.665 DEBUG:boto:path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.665 DEBUG:boto:auth_path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.665 DEBUG:boto:Method: GET 2024-04-09T21:46:59.665 DEBUG:boto:Path: /xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.665 DEBUG:boto:Data: 2024-04-09T21:46:59.665 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.665 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.665 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.665 DEBUG:boto:Params: {} 2024-04-09T21:46:59.666 DEBUG:boto:Token: None 2024-04-09T21:46:59.666 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.666 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfa53cfa33fb45cd-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.668 DEBUG:boto:b'xryqqk-7myobj1000falsemyobj2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.669 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.669 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.669 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.669 DEBUG:boto:Path: /xryqqk-7/myobj 2024-04-09T21:46:59.669 DEBUG:boto:Data: 2024-04-09T21:46:59.669 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.669 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.669 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.669 DEBUG:boto:Params: {} 2024-04-09T21:46:59.669 DEBUG:boto:Token: None 2024-04-09T21:46:59.670 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj 2024-04-09T21:46:59.670 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VQHotkWEzbSF590ePUilQ5N8yIo= 2024-04-09T21:46:59.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VQHotkWEzbSF590ePUilQ5N8yIo=', 'Content-Length': '0'} 2024-04-09T21:46:59.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee320c1d76ae5f7d-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.672 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.672 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.672 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.672 DEBUG:boto:Path: /xryqqk-7/myobj 2024-04-09T21:46:59.672 DEBUG:boto:Data: 2024-04-09T21:46:59.672 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.672 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.672 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.672 DEBUG:boto:Params: {} 2024-04-09T21:46:59.673 DEBUG:boto:Token: None 2024-04-09T21:46:59.673 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj 2024-04-09T21:46:59.673 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VQHotkWEzbSF590ePUilQ5N8yIo= 2024-04-09T21:46:59.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VQHotkWEzbSF590ePUilQ5N8yIo=', 'Content-Length': '0'} 2024-04-09T21:46:59.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7f3f96e652d6580-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:46:59.675 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.675 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.675 DEBUG:boto:Method: GET 2024-04-09T21:46:59.675 DEBUG:boto:Path: /xryqqk-7/myobj 2024-04-09T21:46:59.675 DEBUG:boto:Data: 2024-04-09T21:46:59.675 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.675 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.675 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.675 DEBUG:boto:Params: {} 2024-04-09T21:46:59.675 DEBUG:boto:Token: None 2024-04-09T21:46:59.675 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj 2024-04-09T21:46:59.676 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/WlFhmnSZYx/S3SMjYSvHoxZOdg= 2024-04-09T21:46:59.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/WlFhmnSZYx/S3SMjYSvHoxZOdg=', 'Content-Length': '0'} 2024-04-09T21:46:59.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-7:a5723776-f142-428c-a6df-d7bf949257da.4372.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090a43c1d5a3ed8ef-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.677 DEBUG:boto:path=/xryqqk-7/myobj 2024-04-09T21:46:59.678 DEBUG:boto:auth_path=/xryqqk-7/myobj 2024-04-09T21:46:59.678 DEBUG:boto:Method: GET 2024-04-09T21:46:59.678 DEBUG:boto:Path: /xryqqk-7/myobj 2024-04-09T21:46:59.678 DEBUG:boto:Data: 2024-04-09T21:46:59.678 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.678 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.678 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.678 DEBUG:boto:Params: {} 2024-04-09T21:46:59.678 DEBUG:boto:Token: None 2024-04-09T21:46:59.678 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/myobj 2024-04-09T21:46:59.679 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/WlFhmnSZYx/S3SMjYSvHoxZOdg= 2024-04-09T21:46:59.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/WlFhmnSZYx/S3SMjYSvHoxZOdg=', 'Content-Length': '0'} 2024-04-09T21:46:59.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091459ec01e5e840d-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.681 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.681 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.681 DEBUG:boto:path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.681 DEBUG:boto:auth_path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.681 DEBUG:boto:Method: GET 2024-04-09T21:46:59.681 DEBUG:boto:Path: /xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.681 DEBUG:boto:Data: 2024-04-09T21:46:59.681 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.682 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.682 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.682 DEBUG:boto:Params: {} 2024-04-09T21:46:59.682 DEBUG:boto:Token: None 2024-04-09T21:46:59.682 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.682 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000385640726be9f5ae-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.685 DEBUG:boto:b'xryqqk-7myobj1000falsemyobj2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.685 DEBUG:boto:path=/xryqqk-7/ 2024-04-09T21:46:59.685 DEBUG:boto:auth_path=/xryqqk-7/ 2024-04-09T21:46:59.685 DEBUG:boto:path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.685 DEBUG:boto:auth_path=/xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.685 DEBUG:boto:Method: GET 2024-04-09T21:46:59.685 DEBUG:boto:Path: /xryqqk-7/?prefix=myobj 2024-04-09T21:46:59.686 DEBUG:boto:Data: 2024-04-09T21:46:59.686 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.686 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.686 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.686 DEBUG:boto:Params: {} 2024-04-09T21:46:59.686 DEBUG:boto:Token: None 2024-04-09T21:46:59.686 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-7/ 2024-04-09T21:46:59.686 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw= 2024-04-09T21:46:59.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQGxMHuRcFJG5k264GYher0FrRw=', 'Content-Length': '0'} 2024-04-09T21:46:59.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005be40d10429f6f4-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.689 DEBUG:boto:b'xryqqk-7myobj1000falsemyobj2024-04-09T21:44:58.699Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.689 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-7 zones={a2, a1} 2024-04-09T21:46:59.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:46:59.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:46:59.690 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:46:59.750 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:46:59.751 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:46:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:59.789+0000 7f3b55fd3780 1 bucket sync caught up with source: 2024-04-09T21:46:59.791 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.7.6, , 00000000001.8.6, , , , 00000000001.5.6, 00000000001.8.6, , ] 2024-04-09T21:46:59.791 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000001.7.6, , 00000000001.8.6, , , , 00000000001.5.6, 00000000001.8.6, , ] 2024-04-09T21:46:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:46:59.789+0000 7f3b55fd3780 0 bucket checkpoint complete 2024-04-09T21:46:59.803 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-8 zones={a1, a2} 2024-04-09T21:46:59.803 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.804 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.804 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.804 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:46:59.804 DEBUG:boto:Data: 2024-04-09T21:46:59.804 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.804 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.804 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.804 DEBUG:boto:Params: {} 2024-04-09T21:46:59.804 DEBUG:boto:Token: None 2024-04-09T21:46:59.804 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/ 2024-04-09T21:46:59.805 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8txqlk5GaUr13lLEr47NfN2c4NQ= 2024-04-09T21:46:59.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8txqlk5GaUr13lLEr47NfN2c4NQ=', 'Content-Length': '0'} 2024-04-09T21:46:59.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac9c49565eb734bf-006615b753-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.807 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.807 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.807 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.807 DEBUG:boto:Path: /xryqqk-8/ 2024-04-09T21:46:59.807 DEBUG:boto:Data: 2024-04-09T21:46:59.807 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.807 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.807 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.808 DEBUG:boto:Params: {} 2024-04-09T21:46:59.808 DEBUG:boto:Token: None 2024-04-09T21:46:59.808 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/ 2024-04-09T21:46:59.808 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8txqlk5GaUr13lLEr47NfN2c4NQ= 2024-04-09T21:46:59.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8txqlk5GaUr13lLEr47NfN2c4NQ=', 'Content-Length': '0'} 2024-04-09T21:46:59.810 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001861fb662ff879ca-006615b753-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:46:59.810 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.810 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.810 DEBUG:boto:path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.810 DEBUG:boto:auth_path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.810 DEBUG:boto:Method: GET 2024-04-09T21:46:59.810 DEBUG:boto:Path: /xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.811 DEBUG:boto:Data: 2024-04-09T21:46:59.811 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.811 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.811 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.811 DEBUG:boto:Params: {} 2024-04-09T21:46:59.811 DEBUG:boto:Token: None 2024-04-09T21:46:59.811 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/?versions 2024-04-09T21:46:59.811 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM= 2024-04-09T21:46:59.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM=', 'Content-Length': '0'} 2024-04-09T21:46:59.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bfa791a2261aaa3-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.813 DEBUG:boto:b'xryqqk-8999false&nulltrue2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.814 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-09T21:46:59.814 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-09T21:46:59.814 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-09T21:46:59.814 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:46:59.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:46:59.814 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.814 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.815 DEBUG:boto:path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.815 DEBUG:boto:auth_path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.815 DEBUG:boto:Method: GET 2024-04-09T21:46:59.815 DEBUG:boto:Path: /xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.815 DEBUG:boto:Data: 2024-04-09T21:46:59.815 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.815 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.815 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.815 DEBUG:boto:Params: {} 2024-04-09T21:46:59.815 DEBUG:boto:Token: None 2024-04-09T21:46:59.816 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/?versions 2024-04-09T21:46:59.816 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM= 2024-04-09T21:46:59.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM=', 'Content-Length': '0'} 2024-04-09T21:46:59.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a8d98eb6247d28-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.818 DEBUG:boto:b'xryqqk-8999false&nulltrue2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-09T21:46:59.818 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-09T21:46:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-09T21:46:59.819 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:46:59.819 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.819 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.819 DEBUG:boto:path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.819 DEBUG:boto:auth_path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.819 DEBUG:boto:Method: GET 2024-04-09T21:46:59.819 DEBUG:boto:Path: /xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.819 DEBUG:boto:Data: 2024-04-09T21:46:59.819 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.819 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.820 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.820 DEBUG:boto:Params: {} 2024-04-09T21:46:59.820 DEBUG:boto:Token: None 2024-04-09T21:46:59.820 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/?versions 2024-04-09T21:46:59.820 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM= 2024-04-09T21:46:59.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM=', 'Content-Length': '0'} 2024-04-09T21:46:59.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000543f499b3cf73676-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.822 DEBUG:boto:b'xryqqk-8999false&nulltrue2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.823 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.823 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.823 DEBUG:boto:path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.823 DEBUG:boto:auth_path=/xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.823 DEBUG:boto:Method: GET 2024-04-09T21:46:59.823 DEBUG:boto:Path: /xryqqk-8/?versions&max-keys=999 2024-04-09T21:46:59.823 DEBUG:boto:Data: 2024-04-09T21:46:59.823 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.824 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.824 DEBUG:boto:Params: {} 2024-04-09T21:46:59.824 DEBUG:boto:Token: None 2024-04-09T21:46:59.824 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/?versions 2024-04-09T21:46:59.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM= 2024-04-09T21:46:59.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3tHxb/4wGhvInPCESzVy1jKSxHM=', 'Content-Length': '0'} 2024-04-09T21:46:59.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001647d00d303a6b8-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.826 DEBUG:boto:b'xryqqk-8999false&nulltrue2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-09T21:46:59.827 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.827 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.827 DEBUG:boto:path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.827 DEBUG:boto:auth_path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.827 DEBUG:boto:Method: GET 2024-04-09T21:46:59.827 DEBUG:boto:Path: /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.827 DEBUG:boto:Data: 2024-04-09T21:46:59.827 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.828 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.828 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.828 DEBUG:boto:Params: {} 2024-04-09T21:46:59.828 DEBUG:boto:Token: None 2024-04-09T21:46:59.828 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.828 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso= 2024-04-09T21:46:59.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso=', 'Content-Length': '0'} 2024-04-09T21:46:59.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058676d85faf29c02-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.870 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.870 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.871 DEBUG:boto:path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.871 DEBUG:boto:auth_path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.871 DEBUG:boto:Method: GET 2024-04-09T21:46:59.871 DEBUG:boto:Path: /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.871 DEBUG:boto:Data: 2024-04-09T21:46:59.871 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.871 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.871 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.871 DEBUG:boto:Params: {} 2024-04-09T21:46:59.871 DEBUG:boto:Token: None 2024-04-09T21:46:59.871 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.872 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso= 2024-04-09T21:46:59.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso=', 'Content-Length': '0'} 2024-04-09T21:46:59.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7bbfcbc2eadfb95-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.873 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.874 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.874 DEBUG:boto:path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.874 DEBUG:boto:auth_path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.874 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.874 DEBUG:boto:Path: /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.874 DEBUG:boto:Data: 2024-04-09T21:46:59.874 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.874 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.874 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.874 DEBUG:boto:Params: {} 2024-04-09T21:46:59.874 DEBUG:boto:Token: None 2024-04-09T21:46:59.875 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.875 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AGIo57+BZo88mgXUJbn+H5nLS0k= 2024-04-09T21:46:59.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AGIo57+BZo88mgXUJbn+H5nLS0k=', 'Content-Length': '0'} 2024-04-09T21:46:59.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009000401e40e48391-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.876 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.877 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.877 DEBUG:boto:path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.877 DEBUG:boto:auth_path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.877 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.877 DEBUG:boto:Path: /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.877 DEBUG:boto:Data: 2024-04-09T21:46:59.877 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.877 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.877 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.877 DEBUG:boto:Params: {} 2024-04-09T21:46:59.877 DEBUG:boto:Token: None 2024-04-09T21:46:59.878 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.878 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AGIo57+BZo88mgXUJbn+H5nLS0k= 2024-04-09T21:46:59.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AGIo57+BZo88mgXUJbn+H5nLS0k=', 'Content-Length': '0'} 2024-04-09T21:46:59.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000800f8976f7d7aea6-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-09T21:46:59.880 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.880 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.880 DEBUG:boto:path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.880 DEBUG:boto:auth_path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.880 DEBUG:boto:Method: GET 2024-04-09T21:46:59.880 DEBUG:boto:Path: /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.880 DEBUG:boto:Data: 2024-04-09T21:46:59.880 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.880 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.881 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.881 DEBUG:boto:Params: {} 2024-04-09T21:46:59.881 DEBUG:boto:Token: None 2024-04-09T21:46:59.881 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.881 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso= 2024-04-09T21:46:59.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso=', 'Content-Length': '0'} 2024-04-09T21:46:59.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cee4ef0774815360-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.883 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.883 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.883 DEBUG:boto:path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.883 DEBUG:boto:auth_path=/xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.883 DEBUG:boto:Method: GET 2024-04-09T21:46:59.883 DEBUG:boto:Path: /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.883 DEBUG:boto:Data: 2024-04-09T21:46:59.883 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.883 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.883 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.883 DEBUG:boto:Params: {} 2024-04-09T21:46:59.884 DEBUG:boto:Token: None 2024-04-09T21:46:59.884 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26?versionId=null 2024-04-09T21:46:59.884 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso= 2024-04-09T21:46:59.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zk18i2OesEhZJV7QeID6w/h+oso=', 'Content-Length': '0'} 2024-04-09T21:46:59.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4146528ddabacb2-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.886 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.886 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.886 DEBUG:boto:path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.886 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.886 DEBUG:boto:Method: GET 2024-04-09T21:46:59.886 DEBUG:boto:Path: /xryqqk-8/?prefix=%26 2024-04-09T21:46:59.886 DEBUG:boto:Data: 2024-04-09T21:46:59.886 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.887 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.887 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.887 DEBUG:boto:Params: {} 2024-04-09T21:46:59.887 DEBUG:boto:Token: None 2024-04-09T21:46:59.887 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/ 2024-04-09T21:46:59.887 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU= 2024-04-09T21:46:59.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU=', 'Content-Length': '0'} 2024-04-09T21:46:59.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008acee137fd116af-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.889 DEBUG:boto:b'xryqqk-8&1000false&2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.890 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.890 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.890 DEBUG:boto:path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.890 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.890 DEBUG:boto:Method: GET 2024-04-09T21:46:59.890 DEBUG:boto:Path: /xryqqk-8/?prefix=%26 2024-04-09T21:46:59.890 DEBUG:boto:Data: 2024-04-09T21:46:59.890 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.890 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.890 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.891 DEBUG:boto:Params: {} 2024-04-09T21:46:59.891 DEBUG:boto:Token: None 2024-04-09T21:46:59.891 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/ 2024-04-09T21:46:59.891 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU= 2024-04-09T21:46:59.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU=', 'Content-Length': '0'} 2024-04-09T21:46:59.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c328f4ee5f5352f-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.893 DEBUG:boto:b'xryqqk-8&1000false&2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.894 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.894 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.894 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.894 DEBUG:boto:Path: /xryqqk-8/%26 2024-04-09T21:46:59.894 DEBUG:boto:Data: 2024-04-09T21:46:59.894 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.894 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.894 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.894 DEBUG:boto:Params: {} 2024-04-09T21:46:59.894 DEBUG:boto:Token: None 2024-04-09T21:46:59.895 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26 2024-04-09T21:46:59.895 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vSsP7R8lpsYNFDIHZFP2gUSD1vg= 2024-04-09T21:46:59.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vSsP7R8lpsYNFDIHZFP2gUSD1vg=', 'Content-Length': '0'} 2024-04-09T21:46:59.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092d567faa5e7a010-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.897 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.897 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.897 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.897 DEBUG:boto:Path: /xryqqk-8/%26 2024-04-09T21:46:59.897 DEBUG:boto:Data: 2024-04-09T21:46:59.897 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.897 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.897 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.897 DEBUG:boto:Params: {} 2024-04-09T21:46:59.897 DEBUG:boto:Token: None 2024-04-09T21:46:59.898 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26 2024-04-09T21:46:59.898 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vSsP7R8lpsYNFDIHZFP2gUSD1vg= 2024-04-09T21:46:59.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vSsP7R8lpsYNFDIHZFP2gUSD1vg=', 'Content-Length': '0'} 2024-04-09T21:46:59.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000719c704564adc15e-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-09T21:46:59.899 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.899 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.900 DEBUG:boto:Method: GET 2024-04-09T21:46:59.900 DEBUG:boto:Path: /xryqqk-8/%26 2024-04-09T21:46:59.900 DEBUG:boto:Data: 2024-04-09T21:46:59.900 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.900 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.900 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.900 DEBUG:boto:Params: {} 2024-04-09T21:46:59.900 DEBUG:boto:Token: None 2024-04-09T21:46:59.900 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26 2024-04-09T21:46:59.900 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QVLtPfrpss8XwKUdNHCOo12Z3kk= 2024-04-09T21:46:59.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QVLtPfrpss8XwKUdNHCOo12Z3kk=', 'Content-Length': '0'} 2024-04-09T21:46:59.902 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c3cafabd77182fc-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.902 DEBUG:boto:path=/xryqqk-8/%26 2024-04-09T21:46:59.903 DEBUG:boto:auth_path=/xryqqk-8/%26 2024-04-09T21:46:59.903 DEBUG:boto:Method: GET 2024-04-09T21:46:59.903 DEBUG:boto:Path: /xryqqk-8/%26 2024-04-09T21:46:59.903 DEBUG:boto:Data: 2024-04-09T21:46:59.903 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.903 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.903 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.903 DEBUG:boto:Params: {} 2024-04-09T21:46:59.903 DEBUG:boto:Token: None 2024-04-09T21:46:59.903 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%26 2024-04-09T21:46:59.903 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QVLtPfrpss8XwKUdNHCOo12Z3kk= 2024-04-09T21:46:59.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QVLtPfrpss8XwKUdNHCOo12Z3kk=', 'Content-Length': '0'} 2024-04-09T21:46:59.905 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000508ac863ccaa721e-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.905 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.906 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.906 DEBUG:boto:path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.906 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.906 DEBUG:boto:Method: GET 2024-04-09T21:46:59.906 DEBUG:boto:Path: /xryqqk-8/?prefix=%26 2024-04-09T21:46:59.906 DEBUG:boto:Data: 2024-04-09T21:46:59.906 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.906 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.906 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.906 DEBUG:boto:Params: {} 2024-04-09T21:46:59.906 DEBUG:boto:Token: None 2024-04-09T21:46:59.907 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/ 2024-04-09T21:46:59.907 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU= 2024-04-09T21:46:59.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU=', 'Content-Length': '0'} 2024-04-09T21:46:59.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073d7f9dc814bce4e-006615b753-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.950 DEBUG:boto:b'xryqqk-8&1000false&2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.951 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:46:59.951 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:46:59.951 DEBUG:boto:path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.951 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%26 2024-04-09T21:46:59.951 DEBUG:boto:Method: GET 2024-04-09T21:46:59.951 DEBUG:boto:Path: /xryqqk-8/?prefix=%26 2024-04-09T21:46:59.951 DEBUG:boto:Data: 2024-04-09T21:46:59.951 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.951 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.951 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.952 DEBUG:boto:Params: {} 2024-04-09T21:46:59.952 DEBUG:boto:Token: None 2024-04-09T21:46:59.952 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/ 2024-04-09T21:46:59.952 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU= 2024-04-09T21:46:59.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vcE/rHgqBswWfGGwkL54SYA82HU=', 'Content-Length': '0'} 2024-04-09T21:46:59.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f90f06ceaef0130-006615b753-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.954 DEBUG:boto:b'xryqqk-8&1000false&2024-04-09T21:44:58.892Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:46:59.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-09T21:46:59.955 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:46:59.955 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:46:59.955 DEBUG:boto:path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.955 DEBUG:boto:auth_path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.955 DEBUG:boto:Method: GET 2024-04-09T21:46:59.955 DEBUG:boto:Path: /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.955 DEBUG:boto:Data: 2024-04-09T21:46:59.955 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.955 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.956 DEBUG:boto:Params: {} 2024-04-09T21:46:59.956 DEBUG:boto:Token: None 2024-04-09T21:46:59.956 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.956 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4= 2024-04-09T21:46:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4=', 'Content-Length': '0'} 2024-04-09T21:46:59.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009dad832fde4be927-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.958 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:46:59.958 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:46:59.958 DEBUG:boto:path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.958 DEBUG:boto:auth_path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.958 DEBUG:boto:Method: GET 2024-04-09T21:46:59.958 DEBUG:boto:Path: /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.958 DEBUG:boto:Data: 2024-04-09T21:46:59.959 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.959 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.959 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.959 DEBUG:boto:Params: {} 2024-04-09T21:46:59.959 DEBUG:boto:Token: None 2024-04-09T21:46:59.959 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.959 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4= 2024-04-09T21:46:59.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4=', 'Content-Length': '0'} 2024-04-09T21:46:59.961 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea782db046220783-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.961 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:46:59.961 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:46:59.961 DEBUG:boto:path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.961 DEBUG:boto:auth_path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.961 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.962 DEBUG:boto:Path: /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.962 DEBUG:boto:Data: 2024-04-09T21:46:59.962 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.962 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.962 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.962 DEBUG:boto:Params: {} 2024-04-09T21:46:59.962 DEBUG:boto:Token: None 2024-04-09T21:46:59.962 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.962 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DHBprHnWFScnQmC5xSwiKfLE1bA= 2024-04-09T21:46:59.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DHBprHnWFScnQmC5xSwiKfLE1bA=', 'Content-Length': '0'} 2024-04-09T21:46:59.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035ac3c6da6b63a80-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.964 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:46:59.964 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:46:59.964 DEBUG:boto:path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.964 DEBUG:boto:auth_path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.964 DEBUG:boto:Method: HEAD 2024-04-09T21:46:59.964 DEBUG:boto:Path: /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.964 DEBUG:boto:Data: 2024-04-09T21:46:59.965 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.965 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.965 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.965 DEBUG:boto:Params: {} 2024-04-09T21:46:59.965 DEBUG:boto:Token: None 2024-04-09T21:46:59.965 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.965 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DHBprHnWFScnQmC5xSwiKfLE1bA= 2024-04-09T21:46:59.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DHBprHnWFScnQmC5xSwiKfLE1bA=', 'Content-Length': '0'} 2024-04-09T21:46:59.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2d41730992f39ba-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-09T21:46:59.967 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:46:59.967 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:46:59.967 DEBUG:boto:path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.968 DEBUG:boto:auth_path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.968 DEBUG:boto:Method: GET 2024-04-09T21:46:59.968 DEBUG:boto:Path: /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.968 DEBUG:boto:Data: 2024-04-09T21:46:59.968 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.968 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:46:59.968 DEBUG:boto:Port: 8000 2024-04-09T21:46:59.968 DEBUG:boto:Params: {} 2024-04-09T21:46:59.968 DEBUG:boto:Token: None 2024-04-09T21:46:59.968 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.969 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4= 2024-04-09T21:46:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4=', 'Content-Length': '0'} 2024-04-09T21:46:59.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3329c27928c0e81-006615b753-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:46:59.971 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:46:59.971 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:46:59.971 DEBUG:boto:path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.971 DEBUG:boto:auth_path=/xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.971 DEBUG:boto:Method: GET 2024-04-09T21:46:59.971 DEBUG:boto:Path: /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.971 DEBUG:boto:Data: 2024-04-09T21:46:59.971 DEBUG:boto:Headers: {} 2024-04-09T21:46:59.971 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:46:59.971 DEBUG:boto:Port: 8001 2024-04-09T21:46:59.971 DEBUG:boto:Params: {} 2024-04-09T21:46:59.972 DEBUG:boto:Token: None 2024-04-09T21:46:59.972 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:46:59 GMT /xryqqk-8/%3A?versionId=null 2024-04-09T21:46:59.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4= 2024-04-09T21:46:59.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:46:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hIF9Jg3+fuv6hjF3rSZuXc9X8Q4=', 'Content-Length': '0'} 2024-04-09T21:46:59.973 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b104fb39b801cde2-006615b753-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:46:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.014 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.014 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.015 DEBUG:boto:path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.015 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.015 DEBUG:boto:Method: GET 2024-04-09T21:47:00.015 DEBUG:boto:Path: /xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.015 DEBUG:boto:Data: 2024-04-09T21:47:00.015 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.015 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.015 DEBUG:boto:Params: {} 2024-04-09T21:47:00.015 DEBUG:boto:Token: None 2024-04-09T21:47:00.016 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.016 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d017f3a0e89b357a-006615b754-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.018 DEBUG:boto:b'xryqqk-8:1000false:2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.019 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.019 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.019 DEBUG:boto:path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.019 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.019 DEBUG:boto:Method: GET 2024-04-09T21:47:00.019 DEBUG:boto:Path: /xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.019 DEBUG:boto:Data: 2024-04-09T21:47:00.019 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.019 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.020 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.020 DEBUG:boto:Params: {} 2024-04-09T21:47:00.020 DEBUG:boto:Token: None 2024-04-09T21:47:00.020 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.020 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc24ff39d1504989-006615b754-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.022 DEBUG:boto:b'xryqqk-8:1000false:2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.022 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:47:00.023 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:47:00.023 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.023 DEBUG:boto:Path: /xryqqk-8/%3A 2024-04-09T21:47:00.023 DEBUG:boto:Data: 2024-04-09T21:47:00.023 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.023 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.023 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.023 DEBUG:boto:Params: {} 2024-04-09T21:47:00.023 DEBUG:boto:Token: None 2024-04-09T21:47:00.024 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/%3A 2024-04-09T21:47:00.024 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wEeF3MjvJoRajMrUe39XywPfoZc= 2024-04-09T21:47:00.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wEeF3MjvJoRajMrUe39XywPfoZc=', 'Content-Length': '0'} 2024-04-09T21:47:00.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047796905c9d77bff-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.026 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:47:00.026 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:47:00.026 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.026 DEBUG:boto:Path: /xryqqk-8/%3A 2024-04-09T21:47:00.026 DEBUG:boto:Data: 2024-04-09T21:47:00.026 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.026 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.026 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.026 DEBUG:boto:Params: {} 2024-04-09T21:47:00.027 DEBUG:boto:Token: None 2024-04-09T21:47:00.027 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/%3A 2024-04-09T21:47:00.027 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wEeF3MjvJoRajMrUe39XywPfoZc= 2024-04-09T21:47:00.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wEeF3MjvJoRajMrUe39XywPfoZc=', 'Content-Length': '0'} 2024-04-09T21:47:00.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3ea6353f2977804-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-09T21:47:00.029 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:47:00.029 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:47:00.029 DEBUG:boto:Method: GET 2024-04-09T21:47:00.029 DEBUG:boto:Path: /xryqqk-8/%3A 2024-04-09T21:47:00.029 DEBUG:boto:Data: 2024-04-09T21:47:00.029 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.029 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.029 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.030 DEBUG:boto:Params: {} 2024-04-09T21:47:00.030 DEBUG:boto:Token: None 2024-04-09T21:47:00.030 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/%3A 2024-04-09T21:47:00.030 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yJwENYPbQmOF1xQSjZ0vCq89P9E= 2024-04-09T21:47:00.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yJwENYPbQmOF1xQSjZ0vCq89P9E=', 'Content-Length': '0'} 2024-04-09T21:47:00.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f7d1a989ec763c5-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.032 DEBUG:boto:path=/xryqqk-8/%3A 2024-04-09T21:47:00.032 DEBUG:boto:auth_path=/xryqqk-8/%3A 2024-04-09T21:47:00.032 DEBUG:boto:Method: GET 2024-04-09T21:47:00.032 DEBUG:boto:Path: /xryqqk-8/%3A 2024-04-09T21:47:00.032 DEBUG:boto:Data: 2024-04-09T21:47:00.032 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.032 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.032 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.033 DEBUG:boto:Params: {} 2024-04-09T21:47:00.033 DEBUG:boto:Token: None 2024-04-09T21:47:00.033 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/%3A 2024-04-09T21:47:00.033 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yJwENYPbQmOF1xQSjZ0vCq89P9E= 2024-04-09T21:47:00.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yJwENYPbQmOF1xQSjZ0vCq89P9E=', 'Content-Length': '0'} 2024-04-09T21:47:00.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005461263c1aa71e7d-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.035 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.035 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.035 DEBUG:boto:path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.035 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.035 DEBUG:boto:Method: GET 2024-04-09T21:47:00.035 DEBUG:boto:Path: /xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.035 DEBUG:boto:Data: 2024-04-09T21:47:00.036 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.036 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.036 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.036 DEBUG:boto:Params: {} 2024-04-09T21:47:00.036 DEBUG:boto:Token: None 2024-04-09T21:47:00.036 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.036 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001431da32d911821b-006615b754-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.039 DEBUG:boto:b'xryqqk-8:1000false:2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.039 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.039 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.039 DEBUG:boto:path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.039 DEBUG:boto:auth_path=/xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.039 DEBUG:boto:Method: GET 2024-04-09T21:47:00.039 DEBUG:boto:Path: /xryqqk-8/?prefix=%3A 2024-04-09T21:47:00.039 DEBUG:boto:Data: 2024-04-09T21:47:00.040 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.040 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.040 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.040 DEBUG:boto:Params: {} 2024-04-09T21:47:00.040 DEBUG:boto:Token: None 2024-04-09T21:47:00.040 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.040 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d8b9681f89e41f-006615b754-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.042 DEBUG:boto:b'xryqqk-8:1000false:2024-04-09T21:44:58.880Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-09T21:47:00.043 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.043 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.043 DEBUG:boto:path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.043 DEBUG:boto:auth_path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.043 DEBUG:boto:Method: GET 2024-04-09T21:47:00.043 DEBUG:boto:Path: /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.043 DEBUG:boto:Data: 2024-04-09T21:47:00.044 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.044 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.044 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.044 DEBUG:boto:Params: {} 2024-04-09T21:47:00.044 DEBUG:boto:Token: None 2024-04-09T21:47:00.044 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I= 2024-04-09T21:47:00.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I=', 'Content-Length': '0'} 2024-04-09T21:47:00.046 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043c6c260ccecdd82-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.086 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.086 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.087 DEBUG:boto:path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.087 DEBUG:boto:auth_path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.087 DEBUG:boto:Method: GET 2024-04-09T21:47:00.087 DEBUG:boto:Path: /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.087 DEBUG:boto:Data: 2024-04-09T21:47:00.087 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.087 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.087 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.087 DEBUG:boto:Params: {} 2024-04-09T21:47:00.087 DEBUG:boto:Token: None 2024-04-09T21:47:00.087 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.088 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I= 2024-04-09T21:47:00.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I=', 'Content-Length': '0'} 2024-04-09T21:47:00.089 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d4300333e79cfa5-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.090 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.090 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.090 DEBUG:boto:path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.090 DEBUG:boto:auth_path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.090 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.090 DEBUG:boto:Path: /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.090 DEBUG:boto:Data: 2024-04-09T21:47:00.090 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.090 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.090 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.090 DEBUG:boto:Params: {} 2024-04-09T21:47:00.091 DEBUG:boto:Token: None 2024-04-09T21:47:00.091 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.091 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M9+SXjndT5J18kfRrQakVKHrcdc= 2024-04-09T21:47:00.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M9+SXjndT5J18kfRrQakVKHrcdc=', 'Content-Length': '0'} 2024-04-09T21:47:00.093 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b924088874cb1c0d-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.093 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.093 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.093 DEBUG:boto:path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.093 DEBUG:boto:auth_path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.093 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.093 DEBUG:boto:Path: /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.093 DEBUG:boto:Data: 2024-04-09T21:47:00.094 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.094 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.094 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.094 DEBUG:boto:Params: {} 2024-04-09T21:47:00.094 DEBUG:boto:Token: None 2024-04-09T21:47:00.094 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.094 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M9+SXjndT5J18kfRrQakVKHrcdc= 2024-04-09T21:47:00.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M9+SXjndT5J18kfRrQakVKHrcdc=', 'Content-Length': '0'} 2024-04-09T21:47:00.096 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a9267be996a1db6-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-09T21:47:00.096 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.096 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.097 DEBUG:boto:path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.097 DEBUG:boto:auth_path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.097 DEBUG:boto:Method: GET 2024-04-09T21:47:00.097 DEBUG:boto:Path: /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.097 DEBUG:boto:Data: 2024-04-09T21:47:00.097 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.097 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.097 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.097 DEBUG:boto:Params: {} 2024-04-09T21:47:00.097 DEBUG:boto:Token: None 2024-04-09T21:47:00.097 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.098 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I= 2024-04-09T21:47:00.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I=', 'Content-Length': '0'} 2024-04-09T21:47:00.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b6debc90d6e5079-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.100 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.100 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.100 DEBUG:boto:path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.100 DEBUG:boto:auth_path=/xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.100 DEBUG:boto:Method: GET 2024-04-09T21:47:00.100 DEBUG:boto:Path: /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.100 DEBUG:boto:Data: 2024-04-09T21:47:00.100 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.100 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.101 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.101 DEBUG:boto:Params: {} 2024-04-09T21:47:00.101 DEBUG:boto:Token: None 2024-04-09T21:47:00.101 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj?versionId=null 2024-04-09T21:47:00.101 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I= 2024-04-09T21:47:00.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AKLQLfSrbW80r1aT5yMIvXmDA5I=', 'Content-Length': '0'} 2024-04-09T21:47:00.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a09ba1221073abc2-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.103 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.103 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.103 DEBUG:boto:path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.104 DEBUG:boto:auth_path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.104 DEBUG:boto:Method: GET 2024-04-09T21:47:00.104 DEBUG:boto:Path: /xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.104 DEBUG:boto:Data: 2024-04-09T21:47:00.104 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.104 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.104 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.104 DEBUG:boto:Params: {} 2024-04-09T21:47:00.104 DEBUG:boto:Token: None 2024-04-09T21:47:00.104 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.105 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007abce63ca46d8ac6-006615b754-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.107 DEBUG:boto:b'xryqqk-8_myobj1000false_myobj2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.107 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.107 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.107 DEBUG:boto:path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.107 DEBUG:boto:auth_path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.108 DEBUG:boto:Method: GET 2024-04-09T21:47:00.108 DEBUG:boto:Path: /xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.108 DEBUG:boto:Data: 2024-04-09T21:47:00.108 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.108 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.108 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.108 DEBUG:boto:Params: {} 2024-04-09T21:47:00.108 DEBUG:boto:Token: None 2024-04-09T21:47:00.108 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.108 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d59783edc6525860-006615b754-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.154 DEBUG:boto:b'xryqqk-8_myobj1000false_myobj2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.155 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.155 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.155 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.155 DEBUG:boto:Path: /xryqqk-8/_myobj 2024-04-09T21:47:00.155 DEBUG:boto:Data: 2024-04-09T21:47:00.155 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.155 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.155 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.155 DEBUG:boto:Params: {} 2024-04-09T21:47:00.155 DEBUG:boto:Token: None 2024-04-09T21:47:00.156 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj 2024-04-09T21:47:00.156 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vZGRxpThBCCECB7J8Z8LDO7vlF4= 2024-04-09T21:47:00.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vZGRxpThBCCECB7J8Z8LDO7vlF4=', 'Content-Length': '0'} 2024-04-09T21:47:00.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b72f6f3439b6149-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.158 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.158 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.158 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.158 DEBUG:boto:Path: /xryqqk-8/_myobj 2024-04-09T21:47:00.158 DEBUG:boto:Data: 2024-04-09T21:47:00.158 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.158 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.159 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.159 DEBUG:boto:Params: {} 2024-04-09T21:47:00.159 DEBUG:boto:Token: None 2024-04-09T21:47:00.159 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj 2024-04-09T21:47:00.159 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vZGRxpThBCCECB7J8Z8LDO7vlF4= 2024-04-09T21:47:00.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vZGRxpThBCCECB7J8Z8LDO7vlF4=', 'Content-Length': '0'} 2024-04-09T21:47:00.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091648e50fbe6f717-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-09T21:47:00.161 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.161 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.161 DEBUG:boto:Method: GET 2024-04-09T21:47:00.161 DEBUG:boto:Path: /xryqqk-8/_myobj 2024-04-09T21:47:00.161 DEBUG:boto:Data: 2024-04-09T21:47:00.161 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.161 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.161 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.162 DEBUG:boto:Params: {} 2024-04-09T21:47:00.162 DEBUG:boto:Token: None 2024-04-09T21:47:00.162 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj 2024-04-09T21:47:00.162 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3Zpq51OtqwdxjuIJT+QiRH2MrG0= 2024-04-09T21:47:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3Zpq51OtqwdxjuIJT+QiRH2MrG0=', 'Content-Length': '0'} 2024-04-09T21:47:00.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5ff15f303fa68b3-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.164 DEBUG:boto:path=/xryqqk-8/_myobj 2024-04-09T21:47:00.164 DEBUG:boto:auth_path=/xryqqk-8/_myobj 2024-04-09T21:47:00.164 DEBUG:boto:Method: GET 2024-04-09T21:47:00.164 DEBUG:boto:Path: /xryqqk-8/_myobj 2024-04-09T21:47:00.164 DEBUG:boto:Data: 2024-04-09T21:47:00.164 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.165 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.165 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.165 DEBUG:boto:Params: {} 2024-04-09T21:47:00.165 DEBUG:boto:Token: None 2024-04-09T21:47:00.165 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/_myobj 2024-04-09T21:47:00.165 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3Zpq51OtqwdxjuIJT+QiRH2MrG0= 2024-04-09T21:47:00.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3Zpq51OtqwdxjuIJT+QiRH2MrG0=', 'Content-Length': '0'} 2024-04-09T21:47:00.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fc6bf290346f7bf-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.167 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.167 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.167 DEBUG:boto:path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.167 DEBUG:boto:auth_path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.167 DEBUG:boto:Method: GET 2024-04-09T21:47:00.167 DEBUG:boto:Path: /xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.167 DEBUG:boto:Data: 2024-04-09T21:47:00.167 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.167 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.168 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.168 DEBUG:boto:Params: {} 2024-04-09T21:47:00.168 DEBUG:boto:Token: None 2024-04-09T21:47:00.168 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.168 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093f75e1cdaac6eb3-006615b754-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.170 DEBUG:boto:b'xryqqk-8_myobj1000false_myobj2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.171 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.171 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.171 DEBUG:boto:path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.171 DEBUG:boto:auth_path=/xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.171 DEBUG:boto:Method: GET 2024-04-09T21:47:00.171 DEBUG:boto:Path: /xryqqk-8/?prefix=_myobj 2024-04-09T21:47:00.171 DEBUG:boto:Data: 2024-04-09T21:47:00.171 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.171 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.171 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.171 DEBUG:boto:Params: {} 2024-04-09T21:47:00.171 DEBUG:boto:Token: None 2024-04-09T21:47:00.172 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.172 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efdb9aa6b5a1afcb-006615b754-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.174 DEBUG:boto:b'xryqqk-8_myobj1000false_myobj2024-04-09T21:44:58.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:00.174 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.175 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.175 DEBUG:boto:path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.175 DEBUG:boto:auth_path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.175 DEBUG:boto:Method: GET 2024-04-09T21:47:00.175 DEBUG:boto:Path: /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.175 DEBUG:boto:Data: 2024-04-09T21:47:00.175 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.175 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.175 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.175 DEBUG:boto:Params: {} 2024-04-09T21:47:00.175 DEBUG:boto:Token: None 2024-04-09T21:47:00.176 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.176 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s= 2024-04-09T21:47:00.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s=', 'Content-Length': '0'} 2024-04-09T21:47:00.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3a65940edc589c9-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.218 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.219 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.219 DEBUG:boto:path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.219 DEBUG:boto:auth_path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.219 DEBUG:boto:Method: GET 2024-04-09T21:47:00.219 DEBUG:boto:Path: /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.220 DEBUG:boto:Data: 2024-04-09T21:47:00.220 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.220 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.220 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.220 DEBUG:boto:Params: {} 2024-04-09T21:47:00.221 DEBUG:boto:Token: None 2024-04-09T21:47:00.221 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.221 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s= 2024-04-09T21:47:00.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s=', 'Content-Length': '0'} 2024-04-09T21:47:00.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009983bebc4ab200ff-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.224 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.224 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.225 DEBUG:boto:path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.225 DEBUG:boto:auth_path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.225 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.225 DEBUG:boto:Path: /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.225 DEBUG:boto:Data: 2024-04-09T21:47:00.226 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.226 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.226 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.226 DEBUG:boto:Params: {} 2024-04-09T21:47:00.227 DEBUG:boto:Token: None 2024-04-09T21:47:00.227 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.227 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZWH/bWcF8A4v5O3FZpF4opcadGc= 2024-04-09T21:47:00.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZWH/bWcF8A4v5O3FZpF4opcadGc=', 'Content-Length': '0'} 2024-04-09T21:47:00.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004195c9a89b7ece13-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.229 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.230 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.230 DEBUG:boto:path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.230 DEBUG:boto:auth_path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.230 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.230 DEBUG:boto:Path: /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.231 DEBUG:boto:Data: 2024-04-09T21:47:00.231 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.231 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.231 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.231 DEBUG:boto:Params: {} 2024-04-09T21:47:00.232 DEBUG:boto:Token: None 2024-04-09T21:47:00.232 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.233 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZWH/bWcF8A4v5O3FZpF4opcadGc= 2024-04-09T21:47:00.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZWH/bWcF8A4v5O3FZpF4opcadGc=', 'Content-Length': '0'} 2024-04-09T21:47:00.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000500943e5e06dab2d-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:00.235 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.235 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.235 DEBUG:boto:path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.235 DEBUG:boto:auth_path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.235 DEBUG:boto:Method: GET 2024-04-09T21:47:00.236 DEBUG:boto:Path: /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.236 DEBUG:boto:Data: 2024-04-09T21:47:00.236 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.236 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.236 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.236 DEBUG:boto:Params: {} 2024-04-09T21:47:00.236 DEBUG:boto:Token: None 2024-04-09T21:47:00.236 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.237 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s= 2024-04-09T21:47:00.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s=', 'Content-Length': '0'} 2024-04-09T21:47:00.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7431c8ad10530f1-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.239 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.239 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.239 DEBUG:boto:path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.239 DEBUG:boto:auth_path=/xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.239 DEBUG:boto:Method: GET 2024-04-09T21:47:00.239 DEBUG:boto:Path: /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.239 DEBUG:boto:Data: 2024-04-09T21:47:00.239 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.239 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.239 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.239 DEBUG:boto:Params: {} 2024-04-09T21:47:00.240 DEBUG:boto:Token: None 2024-04-09T21:47:00.240 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj?versionId=null 2024-04-09T21:47:00.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s= 2024-04-09T21:47:00.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Oo1nTsaCj6goOAhGfwQXvBGxu/s=', 'Content-Length': '0'} 2024-04-09T21:47:00.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096f325218e4b468f-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.282 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.282 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.283 DEBUG:boto:path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.283 DEBUG:boto:auth_path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.283 DEBUG:boto:Method: GET 2024-04-09T21:47:00.283 DEBUG:boto:Path: /xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.283 DEBUG:boto:Data: 2024-04-09T21:47:00.283 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.283 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.283 DEBUG:boto:Params: {} 2024-04-09T21:47:00.283 DEBUG:boto:Token: None 2024-04-09T21:47:00.283 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.284 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002f6d8b02c3cf790-006615b754-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.286 DEBUG:boto:b'xryqqk-8myobj1000falsemyobj2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.286 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.286 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.287 DEBUG:boto:path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.287 DEBUG:boto:auth_path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.287 DEBUG:boto:Method: GET 2024-04-09T21:47:00.287 DEBUG:boto:Path: /xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.287 DEBUG:boto:Data: 2024-04-09T21:47:00.287 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.287 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.287 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.287 DEBUG:boto:Params: {} 2024-04-09T21:47:00.287 DEBUG:boto:Token: None 2024-04-09T21:47:00.288 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.288 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60bd6bece52ea32-006615b754-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.290 DEBUG:boto:b'xryqqk-8myobj1000falsemyobj2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.290 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.290 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.290 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.291 DEBUG:boto:Path: /xryqqk-8/myobj 2024-04-09T21:47:00.291 DEBUG:boto:Data: 2024-04-09T21:47:00.291 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.291 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.291 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.291 DEBUG:boto:Params: {} 2024-04-09T21:47:00.291 DEBUG:boto:Token: None 2024-04-09T21:47:00.291 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj 2024-04-09T21:47:00.291 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YNZRtNoU0Vjf2KYjzIcskhLSBaY= 2024-04-09T21:47:00.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YNZRtNoU0Vjf2KYjzIcskhLSBaY=', 'Content-Length': '0'} 2024-04-09T21:47:00.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7a24d550280994d-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.293 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.293 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.294 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.294 DEBUG:boto:Path: /xryqqk-8/myobj 2024-04-09T21:47:00.294 DEBUG:boto:Data: 2024-04-09T21:47:00.294 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.294 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.294 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.294 DEBUG:boto:Params: {} 2024-04-09T21:47:00.294 DEBUG:boto:Token: None 2024-04-09T21:47:00.294 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj 2024-04-09T21:47:00.294 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YNZRtNoU0Vjf2KYjzIcskhLSBaY= 2024-04-09T21:47:00.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YNZRtNoU0Vjf2KYjzIcskhLSBaY=', 'Content-Length': '0'} 2024-04-09T21:47:00.296 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096a9b925e6d05d11-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:00.296 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.296 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.297 DEBUG:boto:Method: GET 2024-04-09T21:47:00.297 DEBUG:boto:Path: /xryqqk-8/myobj 2024-04-09T21:47:00.297 DEBUG:boto:Data: 2024-04-09T21:47:00.297 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.297 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.297 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.297 DEBUG:boto:Params: {} 2024-04-09T21:47:00.297 DEBUG:boto:Token: None 2024-04-09T21:47:00.297 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj 2024-04-09T21:47:00.297 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:N1uEh9wxolrlp+2wlCJhtFcycq0= 2024-04-09T21:47:00.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:N1uEh9wxolrlp+2wlCJhtFcycq0=', 'Content-Length': '0'} 2024-04-09T21:47:00.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-8:a5723776-f142-428c-a6df-d7bf949257da.4372.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000542ff74ce16bc38-006615b754-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.299 DEBUG:boto:path=/xryqqk-8/myobj 2024-04-09T21:47:00.299 DEBUG:boto:auth_path=/xryqqk-8/myobj 2024-04-09T21:47:00.299 DEBUG:boto:Method: GET 2024-04-09T21:47:00.299 DEBUG:boto:Path: /xryqqk-8/myobj 2024-04-09T21:47:00.299 DEBUG:boto:Data: 2024-04-09T21:47:00.300 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.300 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.300 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.300 DEBUG:boto:Params: {} 2024-04-09T21:47:00.300 DEBUG:boto:Token: None 2024-04-09T21:47:00.300 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/myobj 2024-04-09T21:47:00.300 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:N1uEh9wxolrlp+2wlCJhtFcycq0= 2024-04-09T21:47:00.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:N1uEh9wxolrlp+2wlCJhtFcycq0=', 'Content-Length': '0'} 2024-04-09T21:47:00.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:44:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d82dd729f25e1e5f-006615b754-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.302 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.302 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.302 DEBUG:boto:path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.302 DEBUG:boto:auth_path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.302 DEBUG:boto:Method: GET 2024-04-09T21:47:00.302 DEBUG:boto:Path: /xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.302 DEBUG:boto:Data: 2024-04-09T21:47:00.303 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.303 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.303 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.303 DEBUG:boto:Params: {} 2024-04-09T21:47:00.303 DEBUG:boto:Token: None 2024-04-09T21:47:00.303 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.303 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfd9d19a0b3bca34-006615b754-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.305 DEBUG:boto:b'xryqqk-8myobj1000falsemyobj2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.306 DEBUG:boto:path=/xryqqk-8/ 2024-04-09T21:47:00.306 DEBUG:boto:auth_path=/xryqqk-8/ 2024-04-09T21:47:00.306 DEBUG:boto:path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.306 DEBUG:boto:auth_path=/xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.306 DEBUG:boto:Method: GET 2024-04-09T21:47:00.306 DEBUG:boto:Path: /xryqqk-8/?prefix=myobj 2024-04-09T21:47:00.306 DEBUG:boto:Data: 2024-04-09T21:47:00.306 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.306 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.307 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.307 DEBUG:boto:Params: {} 2024-04-09T21:47:00.307 DEBUG:boto:Token: None 2024-04-09T21:47:00.307 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-8/ 2024-04-09T21:47:00.307 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M= 2024-04-09T21:47:00.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SL+HJjTOptyVN1g3NdpOLKFIJ7M=', 'Content-Length': '0'} 2024-04-09T21:47:00.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31a796551c8ccbe-006615b754-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.309 DEBUG:boto:b'xryqqk-8myobj1000falsemyobj2024-04-09T21:44:58.820Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:00.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-8 zones={a1, a2} 2024-04-09T21:47:00.310 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-04-09T21:47:00.311 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-9 2024-04-09T21:47:00.311 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:00.311 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:00.311 DEBUG:boto:Method: PUT 2024-04-09T21:47:00.312 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:00.312 DEBUG:boto:Data: 2024-04-09T21:47:00.312 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.312 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.312 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.312 DEBUG:boto:Params: {} 2024-04-09T21:47:00.312 DEBUG:boto:Token: None 2024-04-09T21:47:00.312 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-9/ 2024-04-09T21:47:00.312 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2rbToWHADGf8lw6VOtH5aSH7Ci8= 2024-04-09T21:47:00.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2rbToWHADGf8lw6VOtH5aSH7Ci8=', 'Content-Length': '0'} 2024-04-09T21:47:00.325 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009301c58e669d6906-006615b754-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.325 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-10 2024-04-09T21:47:00.325 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:00.326 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:00.326 DEBUG:boto:Method: PUT 2024-04-09T21:47:00.326 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:47:00.326 DEBUG:boto:Data: 2024-04-09T21:47:00.326 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.326 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.326 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.326 DEBUG:boto:Params: {} 2024-04-09T21:47:00.326 DEBUG:boto:Token: None 2024-04-09T21:47:00.326 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-10/ 2024-04-09T21:47:00.327 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:NTB/4dtNyh6FAmXLeGsDbUmj+Ik= 2024-04-09T21:47:00.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NTB/4dtNyh6FAmXLeGsDbUmj+Ik=', 'Content-Length': '0'} 2024-04-09T21:47:00.359 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a32ebf92178dcac6-006615b754-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.360 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:00.360 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:00.360 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.360 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:00.360 DEBUG:boto:Data: 2024-04-09T21:47:00.360 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.360 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.360 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.360 DEBUG:boto:Params: {} 2024-04-09T21:47:00.360 DEBUG:boto:Token: None 2024-04-09T21:47:00.361 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-9/ 2024-04-09T21:47:00.361 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:InJ2MhWhQMtrEiLPTlABaNHmQd4= 2024-04-09T21:47:00.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:InJ2MhWhQMtrEiLPTlABaNHmQd4=', 'Content-Length': '0'} 2024-04-09T21:47:00.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b574352ceaeb5c0-006615b754-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.363 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:00.363 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:00.363 DEBUG:boto:Method: PUT 2024-04-09T21:47:00.363 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:00.364 DEBUG:boto:Data: 2024-04-09T21:47:00.364 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:47:00.364 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:00.364 DEBUG:boto:Port: 8000 2024-04-09T21:47:00.364 DEBUG:boto:Params: {} 2024-04-09T21:47:00.364 DEBUG:boto:Token: None 2024-04-09T21:47:00.364 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-9/myobj 2024-04-09T21:47:00.364 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PL5Htc4z4gfz1OosdPejb+BwReA= 2024-04-09T21:47:00.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PL5Htc4z4gfz1OosdPejb+BwReA='} 2024-04-09T21:47:00.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c910a63ba3e3de6d-006615b754-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.407 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:00.407 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:00.407 DEBUG:boto:Method: HEAD 2024-04-09T21:47:00.407 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:47:00.407 DEBUG:boto:Data: 2024-04-09T21:47:00.407 DEBUG:boto:Headers: {} 2024-04-09T21:47:00.407 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.408 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.408 DEBUG:boto:Params: {} 2024-04-09T21:47:00.408 DEBUG:boto:Token: None 2024-04-09T21:47:00.408 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-10/ 2024-04-09T21:47:00.408 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EZTln7QTNloAdc9+orwy+Dp0/+g= 2024-04-09T21:47:00.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EZTln7QTNloAdc9+orwy+Dp0/+g=', 'Content-Length': '0'} 2024-04-09T21:47:00.410 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f58ef27c216b59db-006615b754-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.411 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:00.411 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:00.411 DEBUG:boto:Method: PUT 2024-04-09T21:47:00.411 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:00.411 DEBUG:boto:Data: 2024-04-09T21:47:00.411 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:47:00.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:00.411 DEBUG:boto:Port: 8001 2024-04-09T21:47:00.411 DEBUG:boto:Params: {} 2024-04-09T21:47:00.412 DEBUG:boto:Token: None 2024-04-09T21:47:00.412 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:47:00 GMT /xryqqk-10/myobj 2024-04-09T21:47:00.412 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kI/zU0cJD5n+ymA8BbjKaBM8Lcs= 2024-04-09T21:47:00.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:47:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kI/zU0cJD5n+ymA8BbjKaBM8Lcs='} 2024-04-09T21:47:00.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cce24de7874b5e10-006615b754-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:00.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:47:00.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:47:00.421 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:47:00.470 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:47:00.470 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:47:00.505 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:47:00.517 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:47:00.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.343778_63.1", 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.343778Z" 2024-04-09T21:47:00.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:47:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T21:47:00.519 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:47:00.519 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:47:00.520 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699220.343778_63.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:47:00.520 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:47:00.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:47:00.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:47:00.521 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:47:00.573 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:47:00.573 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:47:00.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:00.602+0000 7fd376ab6780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:47:00.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:00.603+0000 7fd376ab6780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.343778_63.1", 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.343778Z", 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:47:00.620 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:47:00.621 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:47:00Z" 2024-04-09T21:47:00.622 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:47:00.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699220.343778_63.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.343778Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:47:00Z" } 2024-04-09T21:47:00.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699220.343778_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.343778Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:47:00.623 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699220.343778_63.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:47:00.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699220.343778_63.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:47:00.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:47:00.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:00.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:00.624 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:47:00.680 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:47:00.680 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:47:00.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:00.713+0000 7ff35c831780 1 waiting to reach incremental sync.. 2024-04-09T21:47:05.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:05.713+0000 7ff35c831780 1 waiting to reach incremental sync.. 2024-04-09T21:47:10.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:10.714+0000 7ff35c831780 1 waiting to reach incremental sync.. 2024-04-09T21:47:15.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:15.715+0000 7ff35c831780 1 waiting to reach incremental sync.. 2024-04-09T21:47:20.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:20.717+0000 7ff35c831780 1 bucket sync caught up with source: 2024-04-09T21:47:20.719 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-09T21:47:20.719 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000001.2.6, , , , , , , , , ] 2024-04-09T21:47:20.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:20.717+0000 7ff35c831780 0 bucket checkpoint complete 2024-04-09T21:47:20.735 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-9 zones={a2, a1} 2024-04-09T21:47:20.735 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.736 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.736 DEBUG:boto:Method: HEAD 2024-04-09T21:47:20.736 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:20.736 DEBUG:boto:Data: 2024-04-09T21:47:20.736 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.736 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.736 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.736 DEBUG:boto:Params: {} 2024-04-09T21:47:20.737 DEBUG:boto:Token: None 2024-04-09T21:47:20.737 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/ 2024-04-09T21:47:20.737 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M6ko39uDTH8kwFdQcnrPxUOuwYQ= 2024-04-09T21:47:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M6ko39uDTH8kwFdQcnrPxUOuwYQ=', 'Content-Length': '0'} 2024-04-09T21:47:20.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1742327999212b7-006615b768-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.740 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.740 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.740 DEBUG:boto:Method: HEAD 2024-04-09T21:47:20.740 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:20.740 DEBUG:boto:Data: 2024-04-09T21:47:20.740 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.740 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.740 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.741 DEBUG:boto:Params: {} 2024-04-09T21:47:20.741 DEBUG:boto:Token: None 2024-04-09T21:47:20.741 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/ 2024-04-09T21:47:20.741 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M6ko39uDTH8kwFdQcnrPxUOuwYQ= 2024-04-09T21:47:20.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M6ko39uDTH8kwFdQcnrPxUOuwYQ=', 'Content-Length': '0'} 2024-04-09T21:47:20.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b28995020307dce6-006615b768-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:47:20.744 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.744 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.744 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.744 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.744 DEBUG:boto:Method: GET 2024-04-09T21:47:20.744 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.744 DEBUG:boto:Data: 2024-04-09T21:47:20.744 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.744 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.744 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.745 DEBUG:boto:Params: {} 2024-04-09T21:47:20.745 DEBUG:boto:Token: None 2024-04-09T21:47:20.745 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/?versions 2024-04-09T21:47:20.745 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs= 2024-04-09T21:47:20.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs=', 'Content-Length': '0'} 2024-04-09T21:47:20.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379c913ab193e865-006615b768-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.790 DEBUG:boto:b'xryqqk-9999falsemyobjnulltrue2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.791 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:47:20.791 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:47:20.791 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.791 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.791 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.791 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.792 DEBUG:boto:Method: GET 2024-04-09T21:47:20.792 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.792 DEBUG:boto:Data: 2024-04-09T21:47:20.792 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.792 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.792 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.792 DEBUG:boto:Params: {} 2024-04-09T21:47:20.792 DEBUG:boto:Token: None 2024-04-09T21:47:20.792 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/?versions 2024-04-09T21:47:20.792 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs= 2024-04-09T21:47:20.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs=', 'Content-Length': '0'} 2024-04-09T21:47:20.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000673bbb54e82257b4-006615b768-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.795 DEBUG:boto:b'xryqqk-9999falsemyobjnulltrue2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.795 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:47:20.795 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.795 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.795 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.795 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.795 DEBUG:boto:Method: GET 2024-04-09T21:47:20.796 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.796 DEBUG:boto:Data: 2024-04-09T21:47:20.796 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.796 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.796 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.796 DEBUG:boto:Params: {} 2024-04-09T21:47:20.796 DEBUG:boto:Token: None 2024-04-09T21:47:20.796 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/?versions 2024-04-09T21:47:20.796 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs= 2024-04-09T21:47:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs=', 'Content-Length': '0'} 2024-04-09T21:47:20.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dcfb445c749fc67-006615b768-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.799 DEBUG:boto:b'xryqqk-9999falsemyobjnulltrue2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.799 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.799 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.799 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.799 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.799 DEBUG:boto:Method: GET 2024-04-09T21:47:20.799 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:20.800 DEBUG:boto:Data: 2024-04-09T21:47:20.800 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.800 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.800 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.800 DEBUG:boto:Params: {} 2024-04-09T21:47:20.800 DEBUG:boto:Token: None 2024-04-09T21:47:20.800 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/?versions 2024-04-09T21:47:20.800 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs= 2024-04-09T21:47:20.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xHEVCrjPHqGMhAHWnFkJMTkN9xs=', 'Content-Length': '0'} 2024-04-09T21:47:20.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fde69bff6680aced-006615b768-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.802 DEBUG:boto:b'xryqqk-9999falsemyobjnulltrue2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:20.803 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.803 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.803 DEBUG:boto:path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.803 DEBUG:boto:auth_path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.803 DEBUG:boto:Method: GET 2024-04-09T21:47:20.803 DEBUG:boto:Path: /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.803 DEBUG:boto:Data: 2024-04-09T21:47:20.803 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.804 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.804 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.804 DEBUG:boto:Params: {} 2024-04-09T21:47:20.804 DEBUG:boto:Token: None 2024-04-09T21:47:20.804 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE= 2024-04-09T21:47:20.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE=', 'Content-Length': '0'} 2024-04-09T21:47:20.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-9:a5723776-f142-428c-a6df-d7bf949257da.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4e083092bf748ea-006615b768-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.806 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.806 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.806 DEBUG:boto:path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.807 DEBUG:boto:auth_path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.807 DEBUG:boto:Method: GET 2024-04-09T21:47:20.807 DEBUG:boto:Path: /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.807 DEBUG:boto:Data: 2024-04-09T21:47:20.807 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.807 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.807 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.807 DEBUG:boto:Params: {} 2024-04-09T21:47:20.807 DEBUG:boto:Token: None 2024-04-09T21:47:20.807 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.807 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE= 2024-04-09T21:47:20.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE=', 'Content-Length': '0'} 2024-04-09T21:47:20.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000537b9765e03451b6-006615b768-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.809 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.810 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.810 DEBUG:boto:path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.810 DEBUG:boto:auth_path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.810 DEBUG:boto:Method: HEAD 2024-04-09T21:47:20.810 DEBUG:boto:Path: /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.810 DEBUG:boto:Data: 2024-04-09T21:47:20.810 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.810 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.810 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.810 DEBUG:boto:Params: {} 2024-04-09T21:47:20.811 DEBUG:boto:Token: None 2024-04-09T21:47:20.811 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.811 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gg7cYTniASbnUXOJXbOiL51BiV4= 2024-04-09T21:47:20.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gg7cYTniASbnUXOJXbOiL51BiV4=', 'Content-Length': '0'} 2024-04-09T21:47:20.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-9:a5723776-f142-428c-a6df-d7bf949257da.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ebb49baf8dfcb4a-006615b768-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.813 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.813 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.813 DEBUG:boto:path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.813 DEBUG:boto:auth_path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.813 DEBUG:boto:Method: HEAD 2024-04-09T21:47:20.813 DEBUG:boto:Path: /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.813 DEBUG:boto:Data: 2024-04-09T21:47:20.813 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.813 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.813 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.814 DEBUG:boto:Params: {} 2024-04-09T21:47:20.814 DEBUG:boto:Token: None 2024-04-09T21:47:20.814 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.814 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gg7cYTniASbnUXOJXbOiL51BiV4= 2024-04-09T21:47:20.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gg7cYTniASbnUXOJXbOiL51BiV4=', 'Content-Length': '0'} 2024-04-09T21:47:20.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a116db6b430215a-006615b768-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:20.816 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.816 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.816 DEBUG:boto:path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.816 DEBUG:boto:auth_path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.816 DEBUG:boto:Method: GET 2024-04-09T21:47:20.817 DEBUG:boto:Path: /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.817 DEBUG:boto:Data: 2024-04-09T21:47:20.817 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.817 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.817 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.817 DEBUG:boto:Params: {} 2024-04-09T21:47:20.817 DEBUG:boto:Token: None 2024-04-09T21:47:20.817 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.817 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE= 2024-04-09T21:47:20.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE=', 'Content-Length': '0'} 2024-04-09T21:47:20.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-9:a5723776-f142-428c-a6df-d7bf949257da.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003950b8a85fd4584b-006615b768-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.819 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.820 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.820 DEBUG:boto:path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.820 DEBUG:boto:auth_path=/xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.820 DEBUG:boto:Method: GET 2024-04-09T21:47:20.820 DEBUG:boto:Path: /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.820 DEBUG:boto:Data: 2024-04-09T21:47:20.820 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.820 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.820 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.820 DEBUG:boto:Params: {} 2024-04-09T21:47:20.820 DEBUG:boto:Token: None 2024-04-09T21:47:20.820 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj?versionId=null 2024-04-09T21:47:20.821 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE= 2024-04-09T21:47:20.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/96wTcEopIezK0HXF0I3P50dJVE=', 'Content-Length': '0'} 2024-04-09T21:47:20.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0ad5aef47a79a25-006615b768-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.823 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.823 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.823 DEBUG:boto:path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.823 DEBUG:boto:auth_path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.823 DEBUG:boto:Method: GET 2024-04-09T21:47:20.823 DEBUG:boto:Path: /xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.823 DEBUG:boto:Data: 2024-04-09T21:47:20.823 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.824 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.824 DEBUG:boto:Params: {} 2024-04-09T21:47:20.824 DEBUG:boto:Token: None 2024-04-09T21:47:20.824 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/ 2024-04-09T21:47:20.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0= 2024-04-09T21:47:20.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0=', 'Content-Length': '0'} 2024-04-09T21:47:20.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000298597f9dd638a27-006615b768-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.870 DEBUG:boto:b'xryqqk-9myobj1000falsemyobj2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.871 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.871 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.871 DEBUG:boto:path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.871 DEBUG:boto:auth_path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.871 DEBUG:boto:Method: GET 2024-04-09T21:47:20.871 DEBUG:boto:Path: /xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.871 DEBUG:boto:Data: 2024-04-09T21:47:20.871 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.871 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.872 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.872 DEBUG:boto:Params: {} 2024-04-09T21:47:20.872 DEBUG:boto:Token: None 2024-04-09T21:47:20.872 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/ 2024-04-09T21:47:20.872 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0= 2024-04-09T21:47:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0=', 'Content-Length': '0'} 2024-04-09T21:47:20.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3444539012b96ab-006615b768-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.875 DEBUG:boto:b'xryqqk-9myobj1000falsemyobj2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.875 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.875 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.875 DEBUG:boto:Method: HEAD 2024-04-09T21:47:20.875 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:20.875 DEBUG:boto:Data: 2024-04-09T21:47:20.875 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.875 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.875 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.876 DEBUG:boto:Params: {} 2024-04-09T21:47:20.876 DEBUG:boto:Token: None 2024-04-09T21:47:20.876 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj 2024-04-09T21:47:20.876 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c5STHhj0FazGAEK9J0EAZTS13KU= 2024-04-09T21:47:20.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c5STHhj0FazGAEK9J0EAZTS13KU=', 'Content-Length': '0'} 2024-04-09T21:47:20.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-9:a5723776-f142-428c-a6df-d7bf949257da.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d2caf2c6c8f19ff-006615b768-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.878 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.878 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.878 DEBUG:boto:Method: HEAD 2024-04-09T21:47:20.878 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:20.878 DEBUG:boto:Data: 2024-04-09T21:47:20.879 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.879 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.879 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.879 DEBUG:boto:Params: {} 2024-04-09T21:47:20.879 DEBUG:boto:Token: None 2024-04-09T21:47:20.879 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj 2024-04-09T21:47:20.879 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c5STHhj0FazGAEK9J0EAZTS13KU= 2024-04-09T21:47:20.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c5STHhj0FazGAEK9J0EAZTS13KU=', 'Content-Length': '0'} 2024-04-09T21:47:20.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d3d989598e34d6a-006615b768-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:20.881 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.881 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.881 DEBUG:boto:Method: GET 2024-04-09T21:47:20.881 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:20.882 DEBUG:boto:Data: 2024-04-09T21:47:20.882 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.882 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.882 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.882 DEBUG:boto:Params: {} 2024-04-09T21:47:20.882 DEBUG:boto:Token: None 2024-04-09T21:47:20.882 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj 2024-04-09T21:47:20.882 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vyODSmYKVRGyE4L2y55c7aUrkhc= 2024-04-09T21:47:20.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vyODSmYKVRGyE4L2y55c7aUrkhc=', 'Content-Length': '0'} 2024-04-09T21:47:20.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-9:a5723776-f142-428c-a6df-d7bf949257da.4372.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be0bfb8c11326c3e-006615b768-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.884 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:20.885 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:20.885 DEBUG:boto:Method: GET 2024-04-09T21:47:20.885 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:20.885 DEBUG:boto:Data: 2024-04-09T21:47:20.885 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.885 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.885 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.885 DEBUG:boto:Params: {} 2024-04-09T21:47:20.885 DEBUG:boto:Token: None 2024-04-09T21:47:20.885 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/myobj 2024-04-09T21:47:20.886 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vyODSmYKVRGyE4L2y55c7aUrkhc= 2024-04-09T21:47:20.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vyODSmYKVRGyE4L2y55c7aUrkhc=', 'Content-Length': '0'} 2024-04-09T21:47:20.887 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000501583b6289f8d59-006615b768-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.930 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.930 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.931 DEBUG:boto:path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.931 DEBUG:boto:auth_path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.931 DEBUG:boto:Method: GET 2024-04-09T21:47:20.931 DEBUG:boto:Path: /xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.931 DEBUG:boto:Data: 2024-04-09T21:47:20.931 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.931 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:20.931 DEBUG:boto:Port: 8001 2024-04-09T21:47:20.931 DEBUG:boto:Params: {} 2024-04-09T21:47:20.931 DEBUG:boto:Token: None 2024-04-09T21:47:20.932 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/ 2024-04-09T21:47:20.932 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0= 2024-04-09T21:47:20.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0=', 'Content-Length': '0'} 2024-04-09T21:47:20.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d45199d4a1519f32-006615b768-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.934 DEBUG:boto:b'xryqqk-9myobj1000falsemyobj2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.935 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:20.935 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:20.935 DEBUG:boto:path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.935 DEBUG:boto:auth_path=/xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.935 DEBUG:boto:Method: GET 2024-04-09T21:47:20.935 DEBUG:boto:Path: /xryqqk-9/?prefix=myobj 2024-04-09T21:47:20.935 DEBUG:boto:Data: 2024-04-09T21:47:20.935 DEBUG:boto:Headers: {} 2024-04-09T21:47:20.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:20.936 DEBUG:boto:Port: 8000 2024-04-09T21:47:20.936 DEBUG:boto:Params: {} 2024-04-09T21:47:20.936 DEBUG:boto:Token: None 2024-04-09T21:47:20.936 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:20 GMT /xryqqk-9/ 2024-04-09T21:47:20.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0= 2024-04-09T21:47:20.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:645dGNCk0VbA3c5KVeAgRfgmdS0=', 'Content-Length': '0'} 2024-04-09T21:47:20.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a64d60542aab8c0d-006615b768-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:20.939 DEBUG:boto:b'xryqqk-9myobj1000falsemyobj2024-04-09T21:47:00.365Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:20.939 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-9 zones={a2, a1} 2024-04-09T21:47:20.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:20.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:20.940 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:47:21.003 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:47:21.122 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:47:21.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:21.041+0000 7f62d46f8780 1 bucket sync caught up with source: 2024-04-09T21:47:21.124 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-04-09T21:47:21.124 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000001.12.6, , , , , , , , , ] 2024-04-09T21:47:21.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:21.041+0000 7f62d46f8780 0 bucket checkpoint complete 2024-04-09T21:47:21.124 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-10 zones={a1, a2} 2024-04-09T21:47:21.124 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.124 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.125 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.125 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:47:21.125 DEBUG:boto:Data: 2024-04-09T21:47:21.125 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.125 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.125 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.125 DEBUG:boto:Params: {} 2024-04-09T21:47:21.125 DEBUG:boto:Token: None 2024-04-09T21:47:21.126 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/ 2024-04-09T21:47:21.126 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:okk3rdMdCGcmuKE0ii+LYL38D+E= 2024-04-09T21:47:21.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:okk3rdMdCGcmuKE0ii+LYL38D+E=', 'Content-Length': '0'} 2024-04-09T21:47:21.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bc6a16f8c070675-006615b769-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.128 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.129 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.129 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.129 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:47:21.129 DEBUG:boto:Data: 2024-04-09T21:47:21.129 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.129 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.129 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.129 DEBUG:boto:Params: {} 2024-04-09T21:47:21.129 DEBUG:boto:Token: None 2024-04-09T21:47:21.129 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/ 2024-04-09T21:47:21.129 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:okk3rdMdCGcmuKE0ii+LYL38D+E= 2024-04-09T21:47:21.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:okk3rdMdCGcmuKE0ii+LYL38D+E=', 'Content-Length': '0'} 2024-04-09T21:47:21.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b90c4d764934d2c1-006615b769-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:47:21.132 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.132 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.132 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.132 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.132 DEBUG:boto:Method: GET 2024-04-09T21:47:21.132 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.132 DEBUG:boto:Data: 2024-04-09T21:47:21.132 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.133 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.133 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.133 DEBUG:boto:Params: {} 2024-04-09T21:47:21.133 DEBUG:boto:Token: None 2024-04-09T21:47:21.133 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/?versions 2024-04-09T21:47:21.133 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc= 2024-04-09T21:47:21.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc=', 'Content-Length': '0'} 2024-04-09T21:47:21.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c534a8e973abaac4-006615b769-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.136 DEBUG:boto:b'xryqqk-10999falsemyobjnulltrue2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.136 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:47:21.136 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:47:21.136 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.137 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.137 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.137 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.137 DEBUG:boto:Method: GET 2024-04-09T21:47:21.137 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.137 DEBUG:boto:Data: 2024-04-09T21:47:21.137 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.137 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.137 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.137 DEBUG:boto:Params: {} 2024-04-09T21:47:21.137 DEBUG:boto:Token: None 2024-04-09T21:47:21.138 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/?versions 2024-04-09T21:47:21.138 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc= 2024-04-09T21:47:21.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc=', 'Content-Length': '0'} 2024-04-09T21:47:21.140 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b258db93062589f-006615b769-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.140 DEBUG:boto:b'xryqqk-10999falsemyobjnulltrue2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.140 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-09T21:47:21.141 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.141 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.141 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.141 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.141 DEBUG:boto:Method: GET 2024-04-09T21:47:21.141 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.141 DEBUG:boto:Data: 2024-04-09T21:47:21.141 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.141 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.141 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.141 DEBUG:boto:Params: {} 2024-04-09T21:47:21.142 DEBUG:boto:Token: None 2024-04-09T21:47:21.142 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/?versions 2024-04-09T21:47:21.142 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc= 2024-04-09T21:47:21.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc=', 'Content-Length': '0'} 2024-04-09T21:47:21.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019577ea299181e1d-006615b769-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.144 DEBUG:boto:b'xryqqk-10999falsemyobjnulltrue2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.144 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.144 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.145 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.145 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.145 DEBUG:boto:Method: GET 2024-04-09T21:47:21.145 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:47:21.145 DEBUG:boto:Data: 2024-04-09T21:47:21.145 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.145 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.145 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.145 DEBUG:boto:Params: {} 2024-04-09T21:47:21.145 DEBUG:boto:Token: None 2024-04-09T21:47:21.146 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/?versions 2024-04-09T21:47:21.146 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc= 2024-04-09T21:47:21.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n3U+dxDS5vSqKuV43dfOXEQTyQc=', 'Content-Length': '0'} 2024-04-09T21:47:21.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e8de59ad18406fa-006615b769-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.148 DEBUG:boto:b'xryqqk-10999falsemyobjnulltrue2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:21.148 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.149 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.149 DEBUG:boto:path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.149 DEBUG:boto:auth_path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.149 DEBUG:boto:Method: GET 2024-04-09T21:47:21.149 DEBUG:boto:Path: /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.149 DEBUG:boto:Data: 2024-04-09T21:47:21.149 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.149 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.149 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.149 DEBUG:boto:Params: {} 2024-04-09T21:47:21.150 DEBUG:boto:Token: None 2024-04-09T21:47:21.150 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.150 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc= 2024-04-09T21:47:21.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc=', 'Content-Length': '0'} 2024-04-09T21:47:21.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-10:a5723776-f142-428c-a6df-d7bf949257da.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbfe6d2e096ce751-006615b769-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.152 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.152 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.152 DEBUG:boto:path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.152 DEBUG:boto:auth_path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.152 DEBUG:boto:Method: GET 2024-04-09T21:47:21.152 DEBUG:boto:Path: /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.152 DEBUG:boto:Data: 2024-04-09T21:47:21.152 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.153 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.153 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.153 DEBUG:boto:Params: {} 2024-04-09T21:47:21.153 DEBUG:boto:Token: None 2024-04-09T21:47:21.153 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.153 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc= 2024-04-09T21:47:21.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc=', 'Content-Length': '0'} 2024-04-09T21:47:21.155 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001dc8dc8fe4edd1d3-006615b769-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.155 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.155 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.155 DEBUG:boto:path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.155 DEBUG:boto:auth_path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.155 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.156 DEBUG:boto:Path: /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.156 DEBUG:boto:Data: 2024-04-09T21:47:21.156 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.156 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.156 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.156 DEBUG:boto:Params: {} 2024-04-09T21:47:21.156 DEBUG:boto:Token: None 2024-04-09T21:47:21.156 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.156 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jtZ292i2BJ3X68GZyoiHv3n08EQ= 2024-04-09T21:47:21.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jtZ292i2BJ3X68GZyoiHv3n08EQ=', 'Content-Length': '0'} 2024-04-09T21:47:21.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-10:a5723776-f142-428c-a6df-d7bf949257da.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082e301aeff7d5b62-006615b769-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.158 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.158 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.159 DEBUG:boto:path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.159 DEBUG:boto:auth_path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.159 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.159 DEBUG:boto:Path: /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.159 DEBUG:boto:Data: 2024-04-09T21:47:21.159 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.159 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.159 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.159 DEBUG:boto:Params: {} 2024-04-09T21:47:21.159 DEBUG:boto:Token: None 2024-04-09T21:47:21.159 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.160 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jtZ292i2BJ3X68GZyoiHv3n08EQ= 2024-04-09T21:47:21.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jtZ292i2BJ3X68GZyoiHv3n08EQ=', 'Content-Length': '0'} 2024-04-09T21:47:21.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cbacbfbe46149f6-006615b769-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:21.161 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.161 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.162 DEBUG:boto:path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.162 DEBUG:boto:auth_path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.162 DEBUG:boto:Method: GET 2024-04-09T21:47:21.162 DEBUG:boto:Path: /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.162 DEBUG:boto:Data: 2024-04-09T21:47:21.162 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.162 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.162 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.162 DEBUG:boto:Params: {} 2024-04-09T21:47:21.162 DEBUG:boto:Token: None 2024-04-09T21:47:21.163 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.163 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc= 2024-04-09T21:47:21.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc=', 'Content-Length': '0'} 2024-04-09T21:47:21.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-10:a5723776-f142-428c-a6df-d7bf949257da.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bf3eb4525e75362-006615b769-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.165 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.165 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.165 DEBUG:boto:path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.165 DEBUG:boto:auth_path=/xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.165 DEBUG:boto:Method: GET 2024-04-09T21:47:21.165 DEBUG:boto:Path: /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.165 DEBUG:boto:Data: 2024-04-09T21:47:21.165 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.165 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.166 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.166 DEBUG:boto:Params: {} 2024-04-09T21:47:21.166 DEBUG:boto:Token: None 2024-04-09T21:47:21.166 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj?versionId=null 2024-04-09T21:47:21.166 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc= 2024-04-09T21:47:21.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dRWyElI+SgNIJfYeJoe07Ju2ohc=', 'Content-Length': '0'} 2024-04-09T21:47:21.168 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8622f8f0abab6a7-006615b769-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.210 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.210 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.211 DEBUG:boto:path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.211 DEBUG:boto:auth_path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.211 DEBUG:boto:Method: GET 2024-04-09T21:47:21.211 DEBUG:boto:Path: /xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.211 DEBUG:boto:Data: 2024-04-09T21:47:21.211 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.211 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.211 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.211 DEBUG:boto:Params: {} 2024-04-09T21:47:21.211 DEBUG:boto:Token: None 2024-04-09T21:47:21.211 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/ 2024-04-09T21:47:21.212 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE= 2024-04-09T21:47:21.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE=', 'Content-Length': '0'} 2024-04-09T21:47:21.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167dcb6ddf767084-006615b769-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.214 DEBUG:boto:b'xryqqk-10myobj1000falsemyobj2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.215 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.215 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.215 DEBUG:boto:path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.215 DEBUG:boto:auth_path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.215 DEBUG:boto:Method: GET 2024-04-09T21:47:21.215 DEBUG:boto:Path: /xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.215 DEBUG:boto:Data: 2024-04-09T21:47:21.215 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.215 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.215 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.215 DEBUG:boto:Params: {} 2024-04-09T21:47:21.216 DEBUG:boto:Token: None 2024-04-09T21:47:21.216 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/ 2024-04-09T21:47:21.216 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE= 2024-04-09T21:47:21.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE=', 'Content-Length': '0'} 2024-04-09T21:47:21.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9dcee7667e6f14d-006615b769-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.218 DEBUG:boto:b'xryqqk-10myobj1000falsemyobj2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.218 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.219 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.219 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.219 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:21.219 DEBUG:boto:Data: 2024-04-09T21:47:21.219 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.219 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.219 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.219 DEBUG:boto:Params: {} 2024-04-09T21:47:21.219 DEBUG:boto:Token: None 2024-04-09T21:47:21.219 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj 2024-04-09T21:47:21.220 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8hPkDGfkGJuDIvHNwHTrLSWqpRo= 2024-04-09T21:47:21.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8hPkDGfkGJuDIvHNwHTrLSWqpRo=', 'Content-Length': '0'} 2024-04-09T21:47:21.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-10:a5723776-f142-428c-a6df-d7bf949257da.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000788216b5673b34d-006615b769-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.221 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.221 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.222 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.222 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:21.222 DEBUG:boto:Data: 2024-04-09T21:47:21.222 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.222 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.222 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.222 DEBUG:boto:Params: {} 2024-04-09T21:47:21.222 DEBUG:boto:Token: None 2024-04-09T21:47:21.222 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj 2024-04-09T21:47:21.223 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8hPkDGfkGJuDIvHNwHTrLSWqpRo= 2024-04-09T21:47:21.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8hPkDGfkGJuDIvHNwHTrLSWqpRo=', 'Content-Length': '0'} 2024-04-09T21:47:21.224 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb4c21f09962eff9-006615b769-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-09T21:47:21.224 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.224 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.224 DEBUG:boto:Method: GET 2024-04-09T21:47:21.225 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:21.225 DEBUG:boto:Data: 2024-04-09T21:47:21.225 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.225 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.225 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.225 DEBUG:boto:Params: {} 2024-04-09T21:47:21.225 DEBUG:boto:Token: None 2024-04-09T21:47:21.225 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj 2024-04-09T21:47:21.225 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5g79SNnhPZ4pCehYPEtJmjSD8Is= 2024-04-09T21:47:21.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5g79SNnhPZ4pCehYPEtJmjSD8Is=', 'Content-Length': '0'} 2024-04-09T21:47:21.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-10:a5723776-f142-428c-a6df-d7bf949257da.4372.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0f702d3eddbbdee-006615b769-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.227 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:21.227 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:21.227 DEBUG:boto:Method: GET 2024-04-09T21:47:21.227 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:21.228 DEBUG:boto:Data: 2024-04-09T21:47:21.228 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.228 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.228 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.228 DEBUG:boto:Params: {} 2024-04-09T21:47:21.228 DEBUG:boto:Token: None 2024-04-09T21:47:21.228 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/myobj 2024-04-09T21:47:21.228 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5g79SNnhPZ4pCehYPEtJmjSD8Is= 2024-04-09T21:47:21.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5g79SNnhPZ4pCehYPEtJmjSD8Is=', 'Content-Length': '0'} 2024-04-09T21:47:21.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ea5b33c069dd404-006615b769-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.230 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.230 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.230 DEBUG:boto:path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.230 DEBUG:boto:auth_path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.230 DEBUG:boto:Method: GET 2024-04-09T21:47:21.231 DEBUG:boto:Path: /xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.231 DEBUG:boto:Data: 2024-04-09T21:47:21.231 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.231 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.231 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.231 DEBUG:boto:Params: {} 2024-04-09T21:47:21.231 DEBUG:boto:Token: None 2024-04-09T21:47:21.231 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/ 2024-04-09T21:47:21.231 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE= 2024-04-09T21:47:21.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE=', 'Content-Length': '0'} 2024-04-09T21:47:21.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5cdcb289c75df00-006615b769-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.274 DEBUG:boto:b'xryqqk-10myobj1000falsemyobj2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.275 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:21.275 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:21.275 DEBUG:boto:path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.275 DEBUG:boto:auth_path=/xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.275 DEBUG:boto:Method: GET 2024-04-09T21:47:21.275 DEBUG:boto:Path: /xryqqk-10/?prefix=myobj 2024-04-09T21:47:21.275 DEBUG:boto:Data: 2024-04-09T21:47:21.275 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.275 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:21.276 DEBUG:boto:Port: 8001 2024-04-09T21:47:21.276 DEBUG:boto:Params: {} 2024-04-09T21:47:21.276 DEBUG:boto:Token: None 2024-04-09T21:47:21.276 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-10/ 2024-04-09T21:47:21.276 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE= 2024-04-09T21:47:21.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OFy+lnlEr21lmJhJnqODxvinJnE=', 'Content-Length': '0'} 2024-04-09T21:47:21.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad5f9aee0a17602-006615b769-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.278 DEBUG:boto:b'xryqqk-10myobj1000falsemyobj2024-04-09T21:47:00.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:47:21.279 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-10 zones={a1, a2} 2024-04-09T21:47:21.279 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:21.279 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:21.279 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.279 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:21.279 DEBUG:boto:Data: 2024-04-09T21:47:21.279 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.279 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.280 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.280 DEBUG:boto:Params: {} 2024-04-09T21:47:21.280 DEBUG:boto:Token: None 2024-04-09T21:47:21.280 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-9/ 2024-04-09T21:47:21.280 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KyG0TUqHzzBrH8HpyYvEEqxOYO4= 2024-04-09T21:47:21.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KyG0TUqHzzBrH8HpyYvEEqxOYO4=', 'Content-Length': '0'} 2024-04-09T21:47:21.282 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d824446789802a9c-006615b769-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.282 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:21.282 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:21.283 DEBUG:boto:Method: HEAD 2024-04-09T21:47:21.283 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:21.283 DEBUG:boto:Data: 2024-04-09T21:47:21.283 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.283 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.283 DEBUG:boto:Params: {} 2024-04-09T21:47:21.283 DEBUG:boto:Token: None 2024-04-09T21:47:21.283 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-9/myobj 2024-04-09T21:47:21.283 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+Z+SY5EsPMwuZG5hpD7TRM3UtGA= 2024-04-09T21:47:21.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+Z+SY5EsPMwuZG5hpD7TRM3UtGA=', 'Content-Length': '0'} 2024-04-09T21:47:21.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6b5932abacf5aa0-006615b769-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.285 DEBUG:boto:path=/xryqqk-9/myobj 2024-04-09T21:47:21.285 DEBUG:boto:auth_path=/xryqqk-9/myobj 2024-04-09T21:47:21.286 DEBUG:boto:Method: DELETE 2024-04-09T21:47:21.286 DEBUG:boto:Path: /xryqqk-9/myobj 2024-04-09T21:47:21.286 DEBUG:boto:Data: 2024-04-09T21:47:21.286 DEBUG:boto:Headers: {} 2024-04-09T21:47:21.286 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:21.286 DEBUG:boto:Port: 8000 2024-04-09T21:47:21.286 DEBUG:boto:Params: {} 2024-04-09T21:47:21.286 DEBUG:boto:Token: None 2024-04-09T21:47:21.286 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:47:21 GMT /xryqqk-9/myobj 2024-04-09T21:47:21.287 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:W7TZhmWCIq9kuXEnmx59eQ/NEqI= 2024-04-09T21:47:21.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:W7TZhmWCIq9kuXEnmx59eQ/NEqI=', 'Content-Length': '0'} 2024-04-09T21:47:21.291 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000232a12c131c2d627-006615b769-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:47:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:21.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:21.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:21.291 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:47:21.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:47:21.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:47:21.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:21.383+0000 7ff8131d7780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:21.385 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-09T21:47:21.385 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.4.6, , , , , , , , , ] 2024-04-09T21:47:26.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:26.385+0000 7ff8131d7780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:26.387 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-09T21:47:26.387 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.4.6, , , , , , , , , ] 2024-04-09T21:47:31.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:31.386+0000 7ff8131d7780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:31.388 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-09T21:47:31.388 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.4.6, , , , , , , , , ] 2024-04-09T21:47:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:36.388+0000 7ff8131d7780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:36.390 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-09T21:47:36.390 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.4.6, , , , , , , , , ] 2024-04-09T21:47:41.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:41.389+0000 7ff8131d7780 1 bucket sync caught up with source: 2024-04-09T21:47:41.391 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000002.4.6, , , , , , , , , ] 2024-04-09T21:47:41.391 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.4.6, , , , , , , , , ] 2024-04-09T21:47:41.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:41.389+0000 7ff8131d7780 0 bucket checkpoint complete 2024-04-09T21:47:41.407 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-9 zones={a2, a1} 2024-04-09T21:47:41.407 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:41.407 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:41.408 DEBUG:boto:Method: HEAD 2024-04-09T21:47:41.408 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:41.408 DEBUG:boto:Data: 2024-04-09T21:47:41.408 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.408 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:41.408 DEBUG:boto:Port: 8001 2024-04-09T21:47:41.408 DEBUG:boto:Params: {} 2024-04-09T21:47:41.409 DEBUG:boto:Token: None 2024-04-09T21:47:41.409 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-9/ 2024-04-09T21:47:41.409 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:W6qMmtqC3tMYFBvWj4CMbp3UMZw= 2024-04-09T21:47:41.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:W6qMmtqC3tMYFBvWj4CMbp3UMZw=', 'Content-Length': '0'} 2024-04-09T21:47:41.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013b3beb9ac8e63fa-006615b77d-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.412 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:41.412 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:41.412 DEBUG:boto:Method: HEAD 2024-04-09T21:47:41.412 DEBUG:boto:Path: /xryqqk-9/ 2024-04-09T21:47:41.412 DEBUG:boto:Data: 2024-04-09T21:47:41.412 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.412 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:41.412 DEBUG:boto:Port: 8000 2024-04-09T21:47:41.413 DEBUG:boto:Params: {} 2024-04-09T21:47:41.413 DEBUG:boto:Token: None 2024-04-09T21:47:41.413 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-9/ 2024-04-09T21:47:41.413 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:W6qMmtqC3tMYFBvWj4CMbp3UMZw= 2024-04-09T21:47:41.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:W6qMmtqC3tMYFBvWj4CMbp3UMZw=', 'Content-Length': '0'} 2024-04-09T21:47:41.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004689693eb494b683-006615b77d-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.415 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:47:41.415 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:41.416 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:41.416 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.416 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.416 DEBUG:boto:Method: GET 2024-04-09T21:47:41.416 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.416 DEBUG:boto:Data: 2024-04-09T21:47:41.416 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.416 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:41.416 DEBUG:boto:Port: 8001 2024-04-09T21:47:41.416 DEBUG:boto:Params: {} 2024-04-09T21:47:41.416 DEBUG:boto:Token: None 2024-04-09T21:47:41.416 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-9/?versions 2024-04-09T21:47:41.417 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4= 2024-04-09T21:47:41.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4=', 'Content-Length': '0'} 2024-04-09T21:47:41.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4e74ed3e836b027-006615b77d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.419 DEBUG:boto:b'xryqqk-9999false' 2024-04-09T21:47:41.420 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:47:41.420 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:41.420 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:41.420 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.420 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.420 DEBUG:boto:Method: GET 2024-04-09T21:47:41.420 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.420 DEBUG:boto:Data: 2024-04-09T21:47:41.420 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.421 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:41.421 DEBUG:boto:Port: 8000 2024-04-09T21:47:41.421 DEBUG:boto:Params: {} 2024-04-09T21:47:41.421 DEBUG:boto:Token: None 2024-04-09T21:47:41.421 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-9/?versions 2024-04-09T21:47:41.421 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4= 2024-04-09T21:47:41.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4=', 'Content-Length': '0'} 2024-04-09T21:47:41.423 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001beb21f3e4580ad1-006615b77d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.424 DEBUG:boto:b'xryqqk-9999false' 2024-04-09T21:47:41.424 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:41.424 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:41.424 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.424 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.424 DEBUG:boto:Method: GET 2024-04-09T21:47:41.424 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.424 DEBUG:boto:Data: 2024-04-09T21:47:41.424 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.425 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:41.425 DEBUG:boto:Port: 8001 2024-04-09T21:47:41.425 DEBUG:boto:Params: {} 2024-04-09T21:47:41.425 DEBUG:boto:Token: None 2024-04-09T21:47:41.425 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-9/?versions 2024-04-09T21:47:41.425 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4= 2024-04-09T21:47:41.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4=', 'Content-Length': '0'} 2024-04-09T21:47:41.427 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004084c94e9966635-006615b77d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.470 DEBUG:boto:b'xryqqk-9999false' 2024-04-09T21:47:41.471 DEBUG:boto:path=/xryqqk-9/ 2024-04-09T21:47:41.471 DEBUG:boto:auth_path=/xryqqk-9/ 2024-04-09T21:47:41.471 DEBUG:boto:path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.471 DEBUG:boto:auth_path=/xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.471 DEBUG:boto:Method: GET 2024-04-09T21:47:41.471 DEBUG:boto:Path: /xryqqk-9/?versions&max-keys=999 2024-04-09T21:47:41.471 DEBUG:boto:Data: 2024-04-09T21:47:41.471 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.471 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:47:41.471 DEBUG:boto:Port: 8000 2024-04-09T21:47:41.472 DEBUG:boto:Params: {} 2024-04-09T21:47:41.472 DEBUG:boto:Token: None 2024-04-09T21:47:41.472 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-9/?versions 2024-04-09T21:47:41.472 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4= 2024-04-09T21:47:41.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:il7ql+SN9UgOYI0LR6/LpwWe1X4=', 'Content-Length': '0'} 2024-04-09T21:47:41.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038292ffc4f76d164-006615b77d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.474 DEBUG:boto:b'xryqqk-9999false' 2024-04-09T21:47:41.474 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-9 zones={a2, a1} 2024-04-09T21:47:41.475 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:47:41.475 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:47:41.475 DEBUG:boto:Method: HEAD 2024-04-09T21:47:41.475 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:47:41.475 DEBUG:boto:Data: 2024-04-09T21:47:41.475 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.475 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:41.475 DEBUG:boto:Port: 8001 2024-04-09T21:47:41.475 DEBUG:boto:Params: {} 2024-04-09T21:47:41.475 DEBUG:boto:Token: None 2024-04-09T21:47:41.475 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-10/ 2024-04-09T21:47:41.476 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/SDEEyv3FLOONiYZntV35d9TjkY= 2024-04-09T21:47:41.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/SDEEyv3FLOONiYZntV35d9TjkY=', 'Content-Length': '0'} 2024-04-09T21:47:41.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b029a3733aafda20-006615b77d-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.478 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:41.478 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:41.478 DEBUG:boto:Method: HEAD 2024-04-09T21:47:41.478 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:41.478 DEBUG:boto:Data: 2024-04-09T21:47:41.478 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.478 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:41.478 DEBUG:boto:Port: 8001 2024-04-09T21:47:41.478 DEBUG:boto:Params: {} 2024-04-09T21:47:41.479 DEBUG:boto:Token: None 2024-04-09T21:47:41.479 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-10/myobj 2024-04-09T21:47:41.479 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gCJqVQ83Ko28wgG7MGWmdV0c3N4= 2024-04-09T21:47:41.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gCJqVQ83Ko28wgG7MGWmdV0c3N4=', 'Content-Length': '0'} 2024-04-09T21:47:41.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:47:00 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029583e7b9b7befe7-006615b77d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.481 DEBUG:boto:path=/xryqqk-10/myobj 2024-04-09T21:47:41.481 DEBUG:boto:auth_path=/xryqqk-10/myobj 2024-04-09T21:47:41.481 DEBUG:boto:Method: DELETE 2024-04-09T21:47:41.481 DEBUG:boto:Path: /xryqqk-10/myobj 2024-04-09T21:47:41.481 DEBUG:boto:Data: 2024-04-09T21:47:41.481 DEBUG:boto:Headers: {} 2024-04-09T21:47:41.481 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:47:41.481 DEBUG:boto:Port: 8001 2024-04-09T21:47:41.481 DEBUG:boto:Params: {} 2024-04-09T21:47:41.481 DEBUG:boto:Token: None 2024-04-09T21:47:41.482 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:47:41 GMT /xryqqk-10/myobj 2024-04-09T21:47:41.482 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:puMThSVpOoCng9R8sAZoxH+pGk4= 2024-04-09T21:47:41.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:47:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:puMThSVpOoCng9R8sAZoxH+pGk4=', 'Content-Length': '0'} 2024-04-09T21:47:41.487 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a78c01e81374fe1c-006615b77d-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:47:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:47:41.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:41.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:47:41.488 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:47:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:47:41.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:47:41.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:41.587+0000 7f8441b79780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:41.589 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-04-09T21:47:41.589 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-04-09T21:47:46.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:46.588+0000 7f8441b79780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:46.590 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-04-09T21:47:46.590 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-04-09T21:47:51.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:51.590+0000 7f8441b79780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:51.592 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-04-09T21:47:51.592 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-04-09T21:47:56.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:47:56.591+0000 7f8441b79780 1 waiting for incremental sync to catch up: 2024-04-09T21:47:56.593 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000001.12.6, , , , , , , , , ] 2024-04-09T21:47:56.593 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-04-09T21:48:01.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:01.592+0000 7f8441b79780 1 bucket sync caught up with source: 2024-04-09T21:48:01.594 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2024-04-09T21:48:01.594 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2024-04-09T21:48:01.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:01.592+0000 7f8441b79780 0 bucket checkpoint complete 2024-04-09T21:48:01.610 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-10 zones={a1, a2} 2024-04-09T21:48:01.611 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:48:01.611 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:48:01.611 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.611 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:48:01.611 DEBUG:boto:Data: 2024-04-09T21:48:01.611 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.611 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.611 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.611 DEBUG:boto:Params: {} 2024-04-09T21:48:01.611 DEBUG:boto:Token: None 2024-04-09T21:48:01.612 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-10/ 2024-04-09T21:48:01.612 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LfFmJ6vClLZHwl+xGjYhWyVHQm0= 2024-04-09T21:48:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LfFmJ6vClLZHwl+xGjYhWyVHQm0=', 'Content-Length': '0'} 2024-04-09T21:48:01.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006369dd8768f7cc3c-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.615 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:48:01.615 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:48:01.615 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.615 DEBUG:boto:Path: /xryqqk-10/ 2024-04-09T21:48:01.615 DEBUG:boto:Data: 2024-04-09T21:48:01.615 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.615 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:01.615 DEBUG:boto:Port: 8001 2024-04-09T21:48:01.615 DEBUG:boto:Params: {} 2024-04-09T21:48:01.615 DEBUG:boto:Token: None 2024-04-09T21:48:01.616 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-10/ 2024-04-09T21:48:01.616 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LfFmJ6vClLZHwl+xGjYhWyVHQm0= 2024-04-09T21:48:01.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LfFmJ6vClLZHwl+xGjYhWyVHQm0=', 'Content-Length': '0'} 2024-04-09T21:48:01.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000500acba8a95770e7-006615b791-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:48:01.618 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:48:01.618 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:48:01.618 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.619 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.619 DEBUG:boto:Method: GET 2024-04-09T21:48:01.619 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.619 DEBUG:boto:Data: 2024-04-09T21:48:01.619 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.619 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.619 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.619 DEBUG:boto:Params: {} 2024-04-09T21:48:01.619 DEBUG:boto:Token: None 2024-04-09T21:48:01.619 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-10/?versions 2024-04-09T21:48:01.620 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE= 2024-04-09T21:48:01.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE=', 'Content-Length': '0'} 2024-04-09T21:48:01.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bb1291e6942456f-006615b791-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.662 DEBUG:boto:b'xryqqk-10999false' 2024-04-09T21:48:01.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:48:01.663 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:48:01.663 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:48:01.663 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.663 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.663 DEBUG:boto:Method: GET 2024-04-09T21:48:01.663 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.663 DEBUG:boto:Data: 2024-04-09T21:48:01.664 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.664 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:01.664 DEBUG:boto:Port: 8001 2024-04-09T21:48:01.664 DEBUG:boto:Params: {} 2024-04-09T21:48:01.664 DEBUG:boto:Token: None 2024-04-09T21:48:01.664 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-10/?versions 2024-04-09T21:48:01.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE= 2024-04-09T21:48:01.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE=', 'Content-Length': '0'} 2024-04-09T21:48:01.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaa8701f861343d8-006615b791-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.667 DEBUG:boto:b'xryqqk-10999false' 2024-04-09T21:48:01.667 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:48:01.667 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:48:01.667 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.667 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.667 DEBUG:boto:Method: GET 2024-04-09T21:48:01.667 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.667 DEBUG:boto:Data: 2024-04-09T21:48:01.668 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.668 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.668 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.668 DEBUG:boto:Params: {} 2024-04-09T21:48:01.668 DEBUG:boto:Token: None 2024-04-09T21:48:01.668 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-10/?versions 2024-04-09T21:48:01.668 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE= 2024-04-09T21:48:01.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE=', 'Content-Length': '0'} 2024-04-09T21:48:01.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bccceecf717c7bb-006615b791-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.671 DEBUG:boto:b'xryqqk-10999false' 2024-04-09T21:48:01.671 DEBUG:boto:path=/xryqqk-10/ 2024-04-09T21:48:01.671 DEBUG:boto:auth_path=/xryqqk-10/ 2024-04-09T21:48:01.671 DEBUG:boto:path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.671 DEBUG:boto:auth_path=/xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.671 DEBUG:boto:Method: GET 2024-04-09T21:48:01.672 DEBUG:boto:Path: /xryqqk-10/?versions&max-keys=999 2024-04-09T21:48:01.672 DEBUG:boto:Data: 2024-04-09T21:48:01.672 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.672 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:01.672 DEBUG:boto:Port: 8001 2024-04-09T21:48:01.672 DEBUG:boto:Params: {} 2024-04-09T21:48:01.672 DEBUG:boto:Token: None 2024-04-09T21:48:01.672 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-10/?versions 2024-04-09T21:48:01.672 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE= 2024-04-09T21:48:01.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WNhlG/MAq+nM7IQ4l35zcanxLnE=', 'Content-Length': '0'} 2024-04-09T21:48:01.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc4affda4e9d92fd-006615b791-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.675 DEBUG:boto:b'xryqqk-10999false' 2024-04-09T21:48:01.675 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-10 zones={a1, a2} 2024-04-09T21:48:01.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-04-09T21:48:01.677 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-11 2024-04-09T21:48:01.677 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.677 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.677 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.677 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.677 DEBUG:boto:Data: 2024-04-09T21:48:01.678 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.678 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.678 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.678 DEBUG:boto:Params: {} 2024-04-09T21:48:01.678 DEBUG:boto:Token: None 2024-04-09T21:48:01.678 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.678 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:10j7qJF13Kx+xwpmJzleXPyHkb4= 2024-04-09T21:48:01.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:10j7qJF13Kx+xwpmJzleXPyHkb4=', 'Content-Length': '0'} 2024-04-09T21:48:01.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d20befe8533abaa6-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.693 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-12 2024-04-09T21:48:01.693 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:01.693 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:01.693 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.693 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:01.693 DEBUG:boto:Data: 2024-04-09T21:48:01.693 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.693 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:01.693 DEBUG:boto:Port: 8001 2024-04-09T21:48:01.694 DEBUG:boto:Params: {} 2024-04-09T21:48:01.694 DEBUG:boto:Token: None 2024-04-09T21:48:01.694 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-12/ 2024-04-09T21:48:01.694 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cojRhJDO97b6Nz4avCW2M+6I+08= 2024-04-09T21:48:01.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cojRhJDO97b6Nz4avCW2M+6I+08=', 'Content-Length': '0'} 2024-04-09T21:48:01.719 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002885a5d51a7e9b6e-006615b791-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.719 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.719 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.719 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.719 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.719 DEBUG:boto:Data: 2024-04-09T21:48:01.719 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.719 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.720 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.720 DEBUG:boto:Params: {} 2024-04-09T21:48:01.720 DEBUG:boto:Token: None 2024-04-09T21:48:01.720 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.720 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025b429c117727ec2-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.722 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:48:01.722 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:48:01.723 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.723 DEBUG:boto:Path: /xryqqk-11/obj1 2024-04-09T21:48:01.723 DEBUG:boto:Data: 2024-04-09T21:48:01.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.723 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.723 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.723 DEBUG:boto:Params: {} 2024-04-09T21:48:01.723 DEBUG:boto:Token: None 2024-04-09T21:48:01.723 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj1 2024-04-09T21:48:01.723 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vpTPsdL1OnbXm7JnjkGJ10cjJrk= 2024-04-09T21:48:01.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vpTPsdL1OnbXm7JnjkGJ10cjJrk='} 2024-04-09T21:48:01.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029326e52b708c8e6-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.767 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.767 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.767 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.767 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.767 DEBUG:boto:Data: 2024-04-09T21:48:01.767 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.767 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.767 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.767 DEBUG:boto:Params: {} 2024-04-09T21:48:01.768 DEBUG:boto:Token: None 2024-04-09T21:48:01.768 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.768 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdf4f4b2c20924c7-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.770 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:48:01.771 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:48:01.771 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.771 DEBUG:boto:Path: /xryqqk-11/obj2 2024-04-09T21:48:01.771 DEBUG:boto:Data: 2024-04-09T21:48:01.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.771 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.771 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.771 DEBUG:boto:Params: {} 2024-04-09T21:48:01.771 DEBUG:boto:Token: None 2024-04-09T21:48:01.771 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj2 2024-04-09T21:48:01.772 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F6NRhpBTU00qVRGFdXggzwPpawQ= 2024-04-09T21:48:01.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F6NRhpBTU00qVRGFdXggzwPpawQ='} 2024-04-09T21:48:01.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f7318b97cec7407-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.777 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.777 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.778 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.778 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.778 DEBUG:boto:Data: 2024-04-09T21:48:01.778 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.778 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.778 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.778 DEBUG:boto:Params: {} 2024-04-09T21:48:01.778 DEBUG:boto:Token: None 2024-04-09T21:48:01.778 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.779 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019e9f18bb43ad2b3-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.781 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:48:01.781 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:48:01.781 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.781 DEBUG:boto:Path: /xryqqk-11/obj3 2024-04-09T21:48:01.781 DEBUG:boto:Data: 2024-04-09T21:48:01.782 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.782 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.782 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.782 DEBUG:boto:Params: {} 2024-04-09T21:48:01.782 DEBUG:boto:Token: None 2024-04-09T21:48:01.782 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj3 2024-04-09T21:48:01.782 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6xiW1cB3GcbfM4RG/Q6GfnztTF0= 2024-04-09T21:48:01.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6xiW1cB3GcbfM4RG/Q6GfnztTF0='} 2024-04-09T21:48:01.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065bf6e16dead069c-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.788 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.789 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.789 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.789 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.789 DEBUG:boto:Data: 2024-04-09T21:48:01.789 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.789 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.789 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.789 DEBUG:boto:Params: {} 2024-04-09T21:48:01.789 DEBUG:boto:Token: None 2024-04-09T21:48:01.790 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.790 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.792 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c993d897ec95c331-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.792 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:48:01.792 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:48:01.792 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.792 DEBUG:boto:Path: /xryqqk-11/obj4 2024-04-09T21:48:01.792 DEBUG:boto:Data: 2024-04-09T21:48:01.792 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.792 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.792 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.792 DEBUG:boto:Params: {} 2024-04-09T21:48:01.793 DEBUG:boto:Token: None 2024-04-09T21:48:01.793 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj4 2024-04-09T21:48:01.793 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e6bxfbZplwvKbpX42mJCsfpfL6w= 2024-04-09T21:48:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e6bxfbZplwvKbpX42mJCsfpfL6w='} 2024-04-09T21:48:01.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce68e4ef577c1ff1-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.835 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.835 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.835 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.835 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.835 DEBUG:boto:Data: 2024-04-09T21:48:01.835 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.835 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.835 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.836 DEBUG:boto:Params: {} 2024-04-09T21:48:01.836 DEBUG:boto:Token: None 2024-04-09T21:48:01.836 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.836 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003e7e23149c6bad1-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.838 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:48:01.839 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:48:01.839 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.839 DEBUG:boto:Path: /xryqqk-11/obj5 2024-04-09T21:48:01.839 DEBUG:boto:Data: 2024-04-09T21:48:01.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.839 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.839 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.839 DEBUG:boto:Params: {} 2024-04-09T21:48:01.839 DEBUG:boto:Token: None 2024-04-09T21:48:01.839 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj5 2024-04-09T21:48:01.840 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KQNYALpg5l7NLKj1K73O1ziRzrI= 2024-04-09T21:48:01.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KQNYALpg5l7NLKj1K73O1ziRzrI='} 2024-04-09T21:48:01.843 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfdee55f85d13fc0-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.844 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.844 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.844 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.844 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.844 DEBUG:boto:Data: 2024-04-09T21:48:01.844 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.844 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.844 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.844 DEBUG:boto:Params: {} 2024-04-09T21:48:01.845 DEBUG:boto:Token: None 2024-04-09T21:48:01.845 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.845 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008aaf64cc574a2269-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.847 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:48:01.847 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:48:01.847 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.848 DEBUG:boto:Path: /xryqqk-11/obj6 2024-04-09T21:48:01.848 DEBUG:boto:Data: 2024-04-09T21:48:01.848 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.848 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.848 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.848 DEBUG:boto:Params: {} 2024-04-09T21:48:01.848 DEBUG:boto:Token: None 2024-04-09T21:48:01.848 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj6 2024-04-09T21:48:01.848 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j+H8oD+bf2pfPcQwiVGhgKVmuuc= 2024-04-09T21:48:01.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j+H8oD+bf2pfPcQwiVGhgKVmuuc='} 2024-04-09T21:48:01.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040627534fe71074f-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.895 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.895 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.895 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.895 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.895 DEBUG:boto:Data: 2024-04-09T21:48:01.895 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.896 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.896 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.896 DEBUG:boto:Params: {} 2024-04-09T21:48:01.896 DEBUG:boto:Token: None 2024-04-09T21:48:01.896 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.896 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fac9dd3bcaae67c-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.898 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:48:01.899 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:48:01.899 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.899 DEBUG:boto:Path: /xryqqk-11/obj7 2024-04-09T21:48:01.899 DEBUG:boto:Data: 2024-04-09T21:48:01.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.899 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.899 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.899 DEBUG:boto:Params: {} 2024-04-09T21:48:01.899 DEBUG:boto:Token: None 2024-04-09T21:48:01.900 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj7 2024-04-09T21:48:01.900 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hir6sQWp0BpbNY538eiI5IMc43E= 2024-04-09T21:48:01.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hir6sQWp0BpbNY538eiI5IMc43E='} 2024-04-09T21:48:01.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c71974859424078-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.903 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.903 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.904 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.904 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.904 DEBUG:boto:Data: 2024-04-09T21:48:01.904 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.904 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.904 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.904 DEBUG:boto:Params: {} 2024-04-09T21:48:01.904 DEBUG:boto:Token: None 2024-04-09T21:48:01.904 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.904 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3860e573e1a5540-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.907 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:48:01.907 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:48:01.907 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.907 DEBUG:boto:Path: /xryqqk-11/obj8 2024-04-09T21:48:01.907 DEBUG:boto:Data: 2024-04-09T21:48:01.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.907 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.907 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.907 DEBUG:boto:Params: {} 2024-04-09T21:48:01.907 DEBUG:boto:Token: None 2024-04-09T21:48:01.908 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj8 2024-04-09T21:48:01.908 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ryve/BpyvJRD6dMxc0+b1uzDbpM= 2024-04-09T21:48:01.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ryve/BpyvJRD6dMxc0+b1uzDbpM='} 2024-04-09T21:48:01.951 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033a87e22a740d5c1-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.951 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.951 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.951 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.951 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.951 DEBUG:boto:Data: 2024-04-09T21:48:01.951 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.951 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.952 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.952 DEBUG:boto:Params: {} 2024-04-09T21:48:01.952 DEBUG:boto:Token: None 2024-04-09T21:48:01.952 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.952 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.954 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f936749b676d1936-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.955 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:48:01.955 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:48:01.955 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.955 DEBUG:boto:Path: /xryqqk-11/obj9 2024-04-09T21:48:01.955 DEBUG:boto:Data: 2024-04-09T21:48:01.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.955 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.955 DEBUG:boto:Params: {} 2024-04-09T21:48:01.955 DEBUG:boto:Token: None 2024-04-09T21:48:01.956 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj9 2024-04-09T21:48:01.956 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PShGQ9TTpFC0PA1qsHHHNBf8yj4= 2024-04-09T21:48:01.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PShGQ9TTpFC0PA1qsHHHNBf8yj4='} 2024-04-09T21:48:01.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040bb5d5196eaf70d-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.961 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:01.961 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:01.961 DEBUG:boto:Method: HEAD 2024-04-09T21:48:01.961 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:01.962 DEBUG:boto:Data: 2024-04-09T21:48:01.962 DEBUG:boto:Headers: {} 2024-04-09T21:48:01.962 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.962 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.962 DEBUG:boto:Params: {} 2024-04-09T21:48:01.962 DEBUG:boto:Token: None 2024-04-09T21:48:01.962 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/ 2024-04-09T21:48:01.962 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-09T21:48:01.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-09T21:48:01.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cff8259818fc95a-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:01.965 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:48:01.965 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:48:01.965 DEBUG:boto:Method: PUT 2024-04-09T21:48:01.965 DEBUG:boto:Path: /xryqqk-11/obj10 2024-04-09T21:48:01.965 DEBUG:boto:Data: 2024-04-09T21:48:01.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:01.965 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:01.965 DEBUG:boto:Port: 8000 2024-04-09T21:48:01.965 DEBUG:boto:Params: {} 2024-04-09T21:48:01.965 DEBUG:boto:Token: None 2024-04-09T21:48:01.966 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:01 GMT /xryqqk-11/obj10 2024-04-09T21:48:01.966 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yQP2Xzi1FWkpJ1AlXgmDZ7Okc4k= 2024-04-09T21:48:01.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQP2Xzi1FWkpJ1AlXgmDZ7Okc4k='} 2024-04-09T21:48:02.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbc754a1c7f13139-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.011 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.011 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.011 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.011 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.011 DEBUG:boto:Data: 2024-04-09T21:48:02.011 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.011 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.012 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.012 DEBUG:boto:Params: {} 2024-04-09T21:48:02.012 DEBUG:boto:Token: None 2024-04-09T21:48:02.012 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.012 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005db5743ef0f4d50-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.014 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:48:02.015 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:48:02.015 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.015 DEBUG:boto:Path: /xryqqk-11/obj11 2024-04-09T21:48:02.015 DEBUG:boto:Data: 2024-04-09T21:48:02.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.015 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.015 DEBUG:boto:Params: {} 2024-04-09T21:48:02.015 DEBUG:boto:Token: None 2024-04-09T21:48:02.015 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj11 2024-04-09T21:48:02.016 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fIRPB5AKIYqRWEKNl9FVFsAvfKo= 2024-04-09T21:48:02.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fIRPB5AKIYqRWEKNl9FVFsAvfKo='} 2024-04-09T21:48:02.020 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c47a5ac167dea7e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.021 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.021 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.021 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.021 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.021 DEBUG:boto:Data: 2024-04-09T21:48:02.021 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.021 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.021 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.021 DEBUG:boto:Params: {} 2024-04-09T21:48:02.021 DEBUG:boto:Token: None 2024-04-09T21:48:02.022 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.022 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.024 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eef1d8e4d40e542-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.024 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:48:02.024 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:48:02.024 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.024 DEBUG:boto:Path: /xryqqk-11/obj12 2024-04-09T21:48:02.024 DEBUG:boto:Data: 2024-04-09T21:48:02.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.025 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.025 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.025 DEBUG:boto:Params: {} 2024-04-09T21:48:02.025 DEBUG:boto:Token: None 2024-04-09T21:48:02.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj12 2024-04-09T21:48:02.025 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jmAvJaBKC44G2ay2WKUMGPR15bw= 2024-04-09T21:48:02.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jmAvJaBKC44G2ay2WKUMGPR15bw='} 2024-04-09T21:48:02.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb8ff2acc0ce2f1f-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.029 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.029 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.029 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.029 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.029 DEBUG:boto:Data: 2024-04-09T21:48:02.029 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.029 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.030 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.030 DEBUG:boto:Params: {} 2024-04-09T21:48:02.030 DEBUG:boto:Token: None 2024-04-09T21:48:02.030 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.030 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b7d25509c46b144-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.032 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:48:02.032 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:48:02.032 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.032 DEBUG:boto:Path: /xryqqk-11/obj13 2024-04-09T21:48:02.033 DEBUG:boto:Data: 2024-04-09T21:48:02.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.033 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.033 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.033 DEBUG:boto:Params: {} 2024-04-09T21:48:02.033 DEBUG:boto:Token: None 2024-04-09T21:48:02.033 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj13 2024-04-09T21:48:02.033 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ffa6RaCpPYGKkNcbTEk37Ok8tsY= 2024-04-09T21:48:02.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ffa6RaCpPYGKkNcbTEk37Ok8tsY='} 2024-04-09T21:48:02.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce2eddf29522a393-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.083 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.083 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.083 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.083 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.083 DEBUG:boto:Data: 2024-04-09T21:48:02.083 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.084 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.084 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.084 DEBUG:boto:Params: {} 2024-04-09T21:48:02.084 DEBUG:boto:Token: None 2024-04-09T21:48:02.084 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.084 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a0bf89b47aee2d2-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.087 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:48:02.087 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:48:02.087 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.087 DEBUG:boto:Path: /xryqqk-11/obj14 2024-04-09T21:48:02.087 DEBUG:boto:Data: 2024-04-09T21:48:02.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.087 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.087 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.087 DEBUG:boto:Params: {} 2024-04-09T21:48:02.088 DEBUG:boto:Token: None 2024-04-09T21:48:02.088 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj14 2024-04-09T21:48:02.088 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZJfB8UL7vvr/mMpI1WWTksH32r8= 2024-04-09T21:48:02.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZJfB8UL7vvr/mMpI1WWTksH32r8='} 2024-04-09T21:48:02.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000408844e562a661fb-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.091 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.092 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.092 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.092 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.092 DEBUG:boto:Data: 2024-04-09T21:48:02.092 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.092 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.092 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.092 DEBUG:boto:Params: {} 2024-04-09T21:48:02.092 DEBUG:boto:Token: None 2024-04-09T21:48:02.092 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.092 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000176d9c305c9c5133-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.095 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:48:02.095 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:48:02.095 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.095 DEBUG:boto:Path: /xryqqk-11/obj15 2024-04-09T21:48:02.095 DEBUG:boto:Data: 2024-04-09T21:48:02.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.095 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.095 DEBUG:boto:Params: {} 2024-04-09T21:48:02.096 DEBUG:boto:Token: None 2024-04-09T21:48:02.096 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj15 2024-04-09T21:48:02.096 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e5GEuEUjNumj/b0l65qcMA3tg38= 2024-04-09T21:48:02.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e5GEuEUjNumj/b0l65qcMA3tg38='} 2024-04-09T21:48:02.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1ee8e4981c8880a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.139 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.139 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.139 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.139 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.139 DEBUG:boto:Data: 2024-04-09T21:48:02.139 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.139 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.139 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.139 DEBUG:boto:Params: {} 2024-04-09T21:48:02.139 DEBUG:boto:Token: None 2024-04-09T21:48:02.140 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.140 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9355467d39f6fd2-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.142 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:48:02.142 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:48:02.142 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.142 DEBUG:boto:Path: /xryqqk-11/obj16 2024-04-09T21:48:02.142 DEBUG:boto:Data: 2024-04-09T21:48:02.142 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.142 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.142 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.142 DEBUG:boto:Params: {} 2024-04-09T21:48:02.143 DEBUG:boto:Token: None 2024-04-09T21:48:02.143 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj16 2024-04-09T21:48:02.143 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QiRHeWodpao5H61ACvGukDEm6TU= 2024-04-09T21:48:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QiRHeWodpao5H61ACvGukDEm6TU='} 2024-04-09T21:48:02.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008a3754e24471d0e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.148 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.148 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.148 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.148 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.148 DEBUG:boto:Data: 2024-04-09T21:48:02.148 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.148 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.148 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.149 DEBUG:boto:Params: {} 2024-04-09T21:48:02.149 DEBUG:boto:Token: None 2024-04-09T21:48:02.149 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.149 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.151 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cff02e3403155ccf-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.152 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:48:02.152 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:48:02.152 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.152 DEBUG:boto:Path: /xryqqk-11/obj17 2024-04-09T21:48:02.152 DEBUG:boto:Data: 2024-04-09T21:48:02.152 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.152 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.152 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.152 DEBUG:boto:Params: {} 2024-04-09T21:48:02.152 DEBUG:boto:Token: None 2024-04-09T21:48:02.153 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj17 2024-04-09T21:48:02.153 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gyfRG+7NLSmPngX/MFTzJAPTJh4= 2024-04-09T21:48:02.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gyfRG+7NLSmPngX/MFTzJAPTJh4='} 2024-04-09T21:48:02.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9b560c3fef51055-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.195 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.195 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.195 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.195 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.195 DEBUG:boto:Data: 2024-04-09T21:48:02.195 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.195 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.196 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.196 DEBUG:boto:Params: {} 2024-04-09T21:48:02.196 DEBUG:boto:Token: None 2024-04-09T21:48:02.196 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.196 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002844b45bcc27a18c-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.198 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:48:02.198 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:48:02.199 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.199 DEBUG:boto:Path: /xryqqk-11/obj18 2024-04-09T21:48:02.199 DEBUG:boto:Data: 2024-04-09T21:48:02.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.199 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.199 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.199 DEBUG:boto:Params: {} 2024-04-09T21:48:02.199 DEBUG:boto:Token: None 2024-04-09T21:48:02.199 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj18 2024-04-09T21:48:02.199 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e5d71+O0LTiXoseyzYABUn0QsMs= 2024-04-09T21:48:02.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e5d71+O0LTiXoseyzYABUn0QsMs='} 2024-04-09T21:48:02.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005ba579102acca70-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.203 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.203 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.203 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.203 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.203 DEBUG:boto:Data: 2024-04-09T21:48:02.204 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.204 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.204 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.204 DEBUG:boto:Params: {} 2024-04-09T21:48:02.204 DEBUG:boto:Token: None 2024-04-09T21:48:02.204 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.204 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eda08343d3b0709-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.206 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:48:02.206 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:48:02.207 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.207 DEBUG:boto:Path: /xryqqk-11/obj19 2024-04-09T21:48:02.207 DEBUG:boto:Data: 2024-04-09T21:48:02.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.207 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.207 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.207 DEBUG:boto:Params: {} 2024-04-09T21:48:02.207 DEBUG:boto:Token: None 2024-04-09T21:48:02.207 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj19 2024-04-09T21:48:02.207 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:NW2pk/My0osbA/0ryq4uV97Stis= 2024-04-09T21:48:02.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NW2pk/My0osbA/0ryq4uV97Stis='} 2024-04-09T21:48:02.212 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc78162f0e13d8ce-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.212 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.212 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.212 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.212 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.212 DEBUG:boto:Data: 2024-04-09T21:48:02.212 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.212 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.212 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.212 DEBUG:boto:Params: {} 2024-04-09T21:48:02.213 DEBUG:boto:Token: None 2024-04-09T21:48:02.213 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.213 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000162f797834e21ebd-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.215 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:48:02.215 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:48:02.215 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.215 DEBUG:boto:Path: /xryqqk-11/obj20 2024-04-09T21:48:02.215 DEBUG:boto:Data: 2024-04-09T21:48:02.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.216 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.216 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.216 DEBUG:boto:Params: {} 2024-04-09T21:48:02.216 DEBUG:boto:Token: None 2024-04-09T21:48:02.216 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj20 2024-04-09T21:48:02.216 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YHyAXfMabPoExf7VVXW1XMqJvFo= 2024-04-09T21:48:02.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YHyAXfMabPoExf7VVXW1XMqJvFo='} 2024-04-09T21:48:02.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080ac41ddd781b20d-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.259 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.259 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.259 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.259 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.259 DEBUG:boto:Data: 2024-04-09T21:48:02.259 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.259 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.259 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.260 DEBUG:boto:Params: {} 2024-04-09T21:48:02.261 DEBUG:boto:Token: None 2024-04-09T21:48:02.261 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.261 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.263 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8947376b201bddc-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.264 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:48:02.264 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:48:02.264 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.264 DEBUG:boto:Path: /xryqqk-11/obj21 2024-04-09T21:48:02.264 DEBUG:boto:Data: 2024-04-09T21:48:02.264 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.264 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.264 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.265 DEBUG:boto:Params: {} 2024-04-09T21:48:02.265 DEBUG:boto:Token: None 2024-04-09T21:48:02.265 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj21 2024-04-09T21:48:02.265 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:boSJVHq3KZZANb35zwwtXHXRoU4= 2024-04-09T21:48:02.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:boSJVHq3KZZANb35zwwtXHXRoU4='} 2024-04-09T21:48:02.269 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f5ff2b8aaf3437f-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.270 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.270 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.270 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.270 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.270 DEBUG:boto:Data: 2024-04-09T21:48:02.270 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.270 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.270 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.270 DEBUG:boto:Params: {} 2024-04-09T21:48:02.271 DEBUG:boto:Token: None 2024-04-09T21:48:02.271 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.271 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078f4224f64225dca-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.273 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:48:02.273 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:48:02.273 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.273 DEBUG:boto:Path: /xryqqk-11/obj22 2024-04-09T21:48:02.273 DEBUG:boto:Data: 2024-04-09T21:48:02.273 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.274 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.274 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.274 DEBUG:boto:Params: {} 2024-04-09T21:48:02.274 DEBUG:boto:Token: None 2024-04-09T21:48:02.274 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj22 2024-04-09T21:48:02.274 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ngCUYHDgGDDwslHo7/zNuoMEIq8= 2024-04-09T21:48:02.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ngCUYHDgGDDwslHo7/zNuoMEIq8='} 2024-04-09T21:48:02.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001af5daba99ff485c-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.319 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.319 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.319 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.319 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.319 DEBUG:boto:Data: 2024-04-09T21:48:02.319 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.319 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.320 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.320 DEBUG:boto:Params: {} 2024-04-09T21:48:02.320 DEBUG:boto:Token: None 2024-04-09T21:48:02.320 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e24826b8d4af5512-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.322 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:48:02.322 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:48:02.322 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.323 DEBUG:boto:Path: /xryqqk-11/obj23 2024-04-09T21:48:02.323 DEBUG:boto:Data: 2024-04-09T21:48:02.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.323 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.323 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.323 DEBUG:boto:Params: {} 2024-04-09T21:48:02.323 DEBUG:boto:Token: None 2024-04-09T21:48:02.323 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj23 2024-04-09T21:48:02.323 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6PgDHHc+Da/EF7VisCsXUKMZY08= 2024-04-09T21:48:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6PgDHHc+Da/EF7VisCsXUKMZY08='} 2024-04-09T21:48:02.327 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ce88978399280c3-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.327 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.327 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.327 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.327 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.327 DEBUG:boto:Data: 2024-04-09T21:48:02.328 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.328 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.328 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.328 DEBUG:boto:Params: {} 2024-04-09T21:48:02.328 DEBUG:boto:Token: None 2024-04-09T21:48:02.328 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.328 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c5fd74d7ee1a620-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.330 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:48:02.330 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:48:02.331 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.331 DEBUG:boto:Path: /xryqqk-11/obj24 2024-04-09T21:48:02.331 DEBUG:boto:Data: 2024-04-09T21:48:02.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.331 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.331 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.331 DEBUG:boto:Params: {} 2024-04-09T21:48:02.331 DEBUG:boto:Token: None 2024-04-09T21:48:02.331 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj24 2024-04-09T21:48:02.331 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:87n9HNK3MVMFp31BYAT9rN/lHRM= 2024-04-09T21:48:02.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:87n9HNK3MVMFp31BYAT9rN/lHRM='} 2024-04-09T21:48:02.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7bf423f5c33230e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.335 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.335 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.335 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.335 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.335 DEBUG:boto:Data: 2024-04-09T21:48:02.335 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.335 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.335 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.336 DEBUG:boto:Params: {} 2024-04-09T21:48:02.336 DEBUG:boto:Token: None 2024-04-09T21:48:02.336 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.336 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c3be54a0fd0781-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.338 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:48:02.338 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:48:02.338 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.338 DEBUG:boto:Path: /xryqqk-11/obj25 2024-04-09T21:48:02.338 DEBUG:boto:Data: 2024-04-09T21:48:02.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.339 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.339 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.339 DEBUG:boto:Params: {} 2024-04-09T21:48:02.339 DEBUG:boto:Token: None 2024-04-09T21:48:02.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj25 2024-04-09T21:48:02.339 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4vMVtiUaefthfLa/RvS/o53P1qc= 2024-04-09T21:48:02.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4vMVtiUaefthfLa/RvS/o53P1qc='} 2024-04-09T21:48:02.382 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007732e9ee026e2b4e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.383 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.383 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.383 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.383 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.383 DEBUG:boto:Data: 2024-04-09T21:48:02.383 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.383 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.384 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.384 DEBUG:boto:Params: {} 2024-04-09T21:48:02.384 DEBUG:boto:Token: None 2024-04-09T21:48:02.384 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.384 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097c5acbe868bd1c9-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.386 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:48:02.387 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:48:02.387 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.387 DEBUG:boto:Path: /xryqqk-11/obj26 2024-04-09T21:48:02.387 DEBUG:boto:Data: 2024-04-09T21:48:02.387 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.387 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.387 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.387 DEBUG:boto:Params: {} 2024-04-09T21:48:02.387 DEBUG:boto:Token: None 2024-04-09T21:48:02.387 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj26 2024-04-09T21:48:02.388 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kr6v4spmRTbO0NB4bCRFoPB/rvg= 2024-04-09T21:48:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kr6v4spmRTbO0NB4bCRFoPB/rvg='} 2024-04-09T21:48:02.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7b0d7f517a49c05-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.393 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.393 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.393 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.394 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.394 DEBUG:boto:Data: 2024-04-09T21:48:02.394 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.394 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.394 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.394 DEBUG:boto:Params: {} 2024-04-09T21:48:02.394 DEBUG:boto:Token: None 2024-04-09T21:48:02.394 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.394 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043b1e9101bb1daa8-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.397 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:48:02.397 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:48:02.397 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.397 DEBUG:boto:Path: /xryqqk-11/obj27 2024-04-09T21:48:02.397 DEBUG:boto:Data: 2024-04-09T21:48:02.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.397 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.397 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.397 DEBUG:boto:Params: {} 2024-04-09T21:48:02.398 DEBUG:boto:Token: None 2024-04-09T21:48:02.398 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj27 2024-04-09T21:48:02.398 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AUPavD9oQE0diThNvjteOZNERQY= 2024-04-09T21:48:02.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AUPavD9oQE0diThNvjteOZNERQY='} 2024-04-09T21:48:02.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000803c044d478ef455-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.402 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.403 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.403 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.403 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.403 DEBUG:boto:Data: 2024-04-09T21:48:02.403 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.403 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.403 DEBUG:boto:Params: {} 2024-04-09T21:48:02.403 DEBUG:boto:Token: None 2024-04-09T21:48:02.404 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.405 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007329547d70306325-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.406 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:48:02.406 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:48:02.406 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.406 DEBUG:boto:Path: /xryqqk-11/obj28 2024-04-09T21:48:02.406 DEBUG:boto:Data: 2024-04-09T21:48:02.406 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.406 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.406 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.407 DEBUG:boto:Params: {} 2024-04-09T21:48:02.407 DEBUG:boto:Token: None 2024-04-09T21:48:02.407 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj28 2024-04-09T21:48:02.407 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mviJHRdWk1Nq01BeZB5o9Ol7UGM= 2024-04-09T21:48:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mviJHRdWk1Nq01BeZB5o9Ol7UGM='} 2024-04-09T21:48:02.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094fd2e749ff76f8a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.410 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.410 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.411 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.411 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.411 DEBUG:boto:Data: 2024-04-09T21:48:02.411 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.411 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.411 DEBUG:boto:Params: {} 2024-04-09T21:48:02.411 DEBUG:boto:Token: None 2024-04-09T21:48:02.411 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.411 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.414 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000969af379d364354a-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.414 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:48:02.414 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:48:02.414 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.415 DEBUG:boto:Path: /xryqqk-11/obj29 2024-04-09T21:48:02.415 DEBUG:boto:Data: 2024-04-09T21:48:02.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.415 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.415 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.415 DEBUG:boto:Params: {} 2024-04-09T21:48:02.415 DEBUG:boto:Token: None 2024-04-09T21:48:02.415 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj29 2024-04-09T21:48:02.415 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Thuz2a40OQgl2zPHs7uEjvOYPMY= 2024-04-09T21:48:02.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Thuz2a40OQgl2zPHs7uEjvOYPMY='} 2024-04-09T21:48:02.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d86754716e3462b-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.458 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.459 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.459 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.459 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.459 DEBUG:boto:Data: 2024-04-09T21:48:02.459 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.459 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.459 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.459 DEBUG:boto:Params: {} 2024-04-09T21:48:02.459 DEBUG:boto:Token: None 2024-04-09T21:48:02.460 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.461 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090b9e7ccf75d4840-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.462 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:48:02.462 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:48:02.462 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.462 DEBUG:boto:Path: /xryqqk-11/obj30 2024-04-09T21:48:02.462 DEBUG:boto:Data: 2024-04-09T21:48:02.462 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.462 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.462 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.462 DEBUG:boto:Params: {} 2024-04-09T21:48:02.463 DEBUG:boto:Token: None 2024-04-09T21:48:02.463 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj30 2024-04-09T21:48:02.463 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dFhywgTfHI1/Qt1mnj/vhMNT87g= 2024-04-09T21:48:02.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dFhywgTfHI1/Qt1mnj/vhMNT87g='} 2024-04-09T21:48:02.468 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001829af83d0dda297-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.468 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.469 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.469 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.469 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.469 DEBUG:boto:Data: 2024-04-09T21:48:02.469 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.469 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.469 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.469 DEBUG:boto:Params: {} 2024-04-09T21:48:02.469 DEBUG:boto:Token: None 2024-04-09T21:48:02.470 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.470 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a236f78904915c2e-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.472 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:48:02.472 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:48:02.472 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.472 DEBUG:boto:Path: /xryqqk-11/obj31 2024-04-09T21:48:02.472 DEBUG:boto:Data: 2024-04-09T21:48:02.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.472 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.472 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.473 DEBUG:boto:Params: {} 2024-04-09T21:48:02.473 DEBUG:boto:Token: None 2024-04-09T21:48:02.473 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj31 2024-04-09T21:48:02.473 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0O9MYwuSQk9BZLQZx2JfrspilSs= 2024-04-09T21:48:02.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0O9MYwuSQk9BZLQZx2JfrspilSs='} 2024-04-09T21:48:02.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c502e6f96f1cf9a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.477 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.477 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.477 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.477 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.477 DEBUG:boto:Data: 2024-04-09T21:48:02.477 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.477 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.477 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.477 DEBUG:boto:Params: {} 2024-04-09T21:48:02.478 DEBUG:boto:Token: None 2024-04-09T21:48:02.478 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.478 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079cfd1d80e75f5a0-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.480 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:48:02.480 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:48:02.480 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.480 DEBUG:boto:Path: /xryqqk-11/obj32 2024-04-09T21:48:02.480 DEBUG:boto:Data: 2024-04-09T21:48:02.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.480 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.481 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.481 DEBUG:boto:Params: {} 2024-04-09T21:48:02.481 DEBUG:boto:Token: None 2024-04-09T21:48:02.481 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj32 2024-04-09T21:48:02.481 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:eOnbU1Pe66azANzN4c6FDGdNyzE= 2024-04-09T21:48:02.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eOnbU1Pe66azANzN4c6FDGdNyzE='} 2024-04-09T21:48:02.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055e8e98684d4b318-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.527 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.527 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.527 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.527 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.527 DEBUG:boto:Data: 2024-04-09T21:48:02.527 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.528 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.528 DEBUG:boto:Params: {} 2024-04-09T21:48:02.528 DEBUG:boto:Token: None 2024-04-09T21:48:02.528 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090ea7c50e167c29a-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.530 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:48:02.531 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:48:02.531 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.531 DEBUG:boto:Path: /xryqqk-11/obj33 2024-04-09T21:48:02.531 DEBUG:boto:Data: 2024-04-09T21:48:02.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.531 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.531 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.531 DEBUG:boto:Params: {} 2024-04-09T21:48:02.531 DEBUG:boto:Token: None 2024-04-09T21:48:02.531 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj33 2024-04-09T21:48:02.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jw3/wO2AO4EACF95Dbeb4KCTIrA= 2024-04-09T21:48:02.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jw3/wO2AO4EACF95Dbeb4KCTIrA='} 2024-04-09T21:48:02.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d59dbd04f7559e0c-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.535 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.535 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.535 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.535 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.535 DEBUG:boto:Data: 2024-04-09T21:48:02.535 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.535 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.536 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.536 DEBUG:boto:Params: {} 2024-04-09T21:48:02.536 DEBUG:boto:Token: None 2024-04-09T21:48:02.536 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000430ed0b07713efe6-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.539 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:48:02.539 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:48:02.539 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.539 DEBUG:boto:Path: /xryqqk-11/obj34 2024-04-09T21:48:02.539 DEBUG:boto:Data: 2024-04-09T21:48:02.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.539 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.539 DEBUG:boto:Params: {} 2024-04-09T21:48:02.540 DEBUG:boto:Token: None 2024-04-09T21:48:02.540 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj34 2024-04-09T21:48:02.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:avRi6ktHqLAqcMbeHFsTRwjcPWU= 2024-04-09T21:48:02.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:avRi6ktHqLAqcMbeHFsTRwjcPWU='} 2024-04-09T21:48:02.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfcdfa9278c6cb7d-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.587 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.587 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.587 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.587 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.587 DEBUG:boto:Data: 2024-04-09T21:48:02.587 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.587 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.587 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.588 DEBUG:boto:Params: {} 2024-04-09T21:48:02.588 DEBUG:boto:Token: None 2024-04-09T21:48:02.588 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.588 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d06d2d5f1dca20bd-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.590 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:48:02.591 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:48:02.591 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.591 DEBUG:boto:Path: /xryqqk-11/obj35 2024-04-09T21:48:02.591 DEBUG:boto:Data: 2024-04-09T21:48:02.591 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.591 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.591 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.591 DEBUG:boto:Params: {} 2024-04-09T21:48:02.591 DEBUG:boto:Token: None 2024-04-09T21:48:02.592 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj35 2024-04-09T21:48:02.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4ReLLjZgacwyARv4VgAPnyMeiD0= 2024-04-09T21:48:02.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4ReLLjZgacwyARv4VgAPnyMeiD0='} 2024-04-09T21:48:02.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a412cbd90bd9c82-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.595 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.595 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.596 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.596 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.596 DEBUG:boto:Data: 2024-04-09T21:48:02.596 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.596 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.596 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.596 DEBUG:boto:Params: {} 2024-04-09T21:48:02.596 DEBUG:boto:Token: None 2024-04-09T21:48:02.596 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.596 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d35acbcc8eac05f-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.598 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:48:02.599 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:48:02.599 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.599 DEBUG:boto:Path: /xryqqk-11/obj36 2024-04-09T21:48:02.599 DEBUG:boto:Data: 2024-04-09T21:48:02.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.599 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.599 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.599 DEBUG:boto:Params: {} 2024-04-09T21:48:02.599 DEBUG:boto:Token: None 2024-04-09T21:48:02.599 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj36 2024-04-09T21:48:02.600 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AR414Hhi2/5cS4Sb1rr2oUH8DWQ= 2024-04-09T21:48:02.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AR414Hhi2/5cS4Sb1rr2oUH8DWQ='} 2024-04-09T21:48:02.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a06dd4409382b30-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.643 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.643 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.643 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.643 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.643 DEBUG:boto:Data: 2024-04-09T21:48:02.643 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.643 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.644 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.644 DEBUG:boto:Params: {} 2024-04-09T21:48:02.644 DEBUG:boto:Token: None 2024-04-09T21:48:02.644 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.644 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.646 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000408f4efe7eda59e2-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.646 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:48:02.647 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:48:02.647 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.647 DEBUG:boto:Path: /xryqqk-11/obj37 2024-04-09T21:48:02.647 DEBUG:boto:Data: 2024-04-09T21:48:02.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.647 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.647 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.647 DEBUG:boto:Params: {} 2024-04-09T21:48:02.647 DEBUG:boto:Token: None 2024-04-09T21:48:02.647 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj37 2024-04-09T21:48:02.648 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fXIrZo6WwTt6RxorL1uCORDQ+2E= 2024-04-09T21:48:02.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fXIrZo6WwTt6RxorL1uCORDQ+2E='} 2024-04-09T21:48:02.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47897bf408656b2-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.651 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.651 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.651 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.652 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.652 DEBUG:boto:Data: 2024-04-09T21:48:02.652 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.652 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.652 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.652 DEBUG:boto:Params: {} 2024-04-09T21:48:02.652 DEBUG:boto:Token: None 2024-04-09T21:48:02.652 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.652 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed8b45a16ee78afa-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.655 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:48:02.655 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:48:02.655 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.655 DEBUG:boto:Path: /xryqqk-11/obj38 2024-04-09T21:48:02.656 DEBUG:boto:Data: 2024-04-09T21:48:02.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.656 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.656 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.656 DEBUG:boto:Params: {} 2024-04-09T21:48:02.656 DEBUG:boto:Token: None 2024-04-09T21:48:02.656 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj38 2024-04-09T21:48:02.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+w71DItf/rAeXU4RXFWO1xFZaA8= 2024-04-09T21:48:02.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+w71DItf/rAeXU4RXFWO1xFZaA8='} 2024-04-09T21:48:02.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e493b245a6f78d3-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.699 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.699 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.699 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.699 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.699 DEBUG:boto:Data: 2024-04-09T21:48:02.699 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.699 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.699 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.699 DEBUG:boto:Params: {} 2024-04-09T21:48:02.700 DEBUG:boto:Token: None 2024-04-09T21:48:02.700 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.700 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ece3e40a97950fa8-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.702 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:48:02.702 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:48:02.702 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.703 DEBUG:boto:Path: /xryqqk-11/obj39 2024-04-09T21:48:02.703 DEBUG:boto:Data: 2024-04-09T21:48:02.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.703 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.703 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.703 DEBUG:boto:Params: {} 2024-04-09T21:48:02.703 DEBUG:boto:Token: None 2024-04-09T21:48:02.703 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj39 2024-04-09T21:48:02.703 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wBKHZL47USiewqblcFyq/hajjCs= 2024-04-09T21:48:02.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wBKHZL47USiewqblcFyq/hajjCs='} 2024-04-09T21:48:02.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a0493d07cf71a2a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.707 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.707 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.707 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.707 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.707 DEBUG:boto:Data: 2024-04-09T21:48:02.707 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.707 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.708 DEBUG:boto:Params: {} 2024-04-09T21:48:02.708 DEBUG:boto:Token: None 2024-04-09T21:48:02.708 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.708 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3e5d89f7e4cf1e9-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.710 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:48:02.710 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:48:02.710 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.710 DEBUG:boto:Path: /xryqqk-11/obj40 2024-04-09T21:48:02.710 DEBUG:boto:Data: 2024-04-09T21:48:02.710 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.711 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.711 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.711 DEBUG:boto:Params: {} 2024-04-09T21:48:02.711 DEBUG:boto:Token: None 2024-04-09T21:48:02.711 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj40 2024-04-09T21:48:02.711 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b/QIDwGtc/jm7yQ5YwgcYcOF/p8= 2024-04-09T21:48:02.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b/QIDwGtc/jm7yQ5YwgcYcOF/p8='} 2024-04-09T21:48:02.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000057534148d87d2ac-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.714 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.715 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.715 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.715 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.715 DEBUG:boto:Data: 2024-04-09T21:48:02.715 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.715 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.715 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.715 DEBUG:boto:Params: {} 2024-04-09T21:48:02.715 DEBUG:boto:Token: None 2024-04-09T21:48:02.715 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.716 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.717 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c4d5c46c49666cc-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.718 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:48:02.718 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:48:02.718 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.718 DEBUG:boto:Path: /xryqqk-11/obj41 2024-04-09T21:48:02.718 DEBUG:boto:Data: 2024-04-09T21:48:02.718 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.718 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.718 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.719 DEBUG:boto:Params: {} 2024-04-09T21:48:02.719 DEBUG:boto:Token: None 2024-04-09T21:48:02.719 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj41 2024-04-09T21:48:02.719 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:V6bjX4JBZqef8h9wYPLW+wCvzu8= 2024-04-09T21:48:02.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:V6bjX4JBZqef8h9wYPLW+wCvzu8='} 2024-04-09T21:48:02.763 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051e0446cedd8e916-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.763 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.763 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.763 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.763 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.763 DEBUG:boto:Data: 2024-04-09T21:48:02.763 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.763 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.764 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.764 DEBUG:boto:Params: {} 2024-04-09T21:48:02.764 DEBUG:boto:Token: None 2024-04-09T21:48:02.764 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.764 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064de43f5e507f52b-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.766 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:48:02.766 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:48:02.767 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.767 DEBUG:boto:Path: /xryqqk-11/obj42 2024-04-09T21:48:02.767 DEBUG:boto:Data: 2024-04-09T21:48:02.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.767 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.767 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.767 DEBUG:boto:Params: {} 2024-04-09T21:48:02.767 DEBUG:boto:Token: None 2024-04-09T21:48:02.767 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj42 2024-04-09T21:48:02.767 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EwV603UV8+D4s4/OBeeLs9g+iwI= 2024-04-09T21:48:02.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwV603UV8+D4s4/OBeeLs9g+iwI='} 2024-04-09T21:48:02.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f77c6e5cb6754de0-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.771 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.772 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.772 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.772 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.772 DEBUG:boto:Data: 2024-04-09T21:48:02.772 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.772 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.772 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.772 DEBUG:boto:Params: {} 2024-04-09T21:48:02.772 DEBUG:boto:Token: None 2024-04-09T21:48:02.772 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.772 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020a436ff779d077c-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.775 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:48:02.775 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:48:02.775 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.775 DEBUG:boto:Path: /xryqqk-11/obj43 2024-04-09T21:48:02.775 DEBUG:boto:Data: 2024-04-09T21:48:02.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.775 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.776 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.776 DEBUG:boto:Params: {} 2024-04-09T21:48:02.776 DEBUG:boto:Token: None 2024-04-09T21:48:02.776 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj43 2024-04-09T21:48:02.776 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9m5oqFeNohkfqkzXJJ9K5oWv4Lo= 2024-04-09T21:48:02.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9m5oqFeNohkfqkzXJJ9K5oWv4Lo='} 2024-04-09T21:48:02.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069b4351f4547eca0-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.819 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.819 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.819 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.819 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.819 DEBUG:boto:Data: 2024-04-09T21:48:02.819 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.819 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.819 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.819 DEBUG:boto:Params: {} 2024-04-09T21:48:02.819 DEBUG:boto:Token: None 2024-04-09T21:48:02.820 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.820 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.822 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009203c8d624dff12c-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.822 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:48:02.822 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:48:02.822 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.822 DEBUG:boto:Path: /xryqqk-11/obj44 2024-04-09T21:48:02.823 DEBUG:boto:Data: 2024-04-09T21:48:02.823 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.823 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.823 DEBUG:boto:Params: {} 2024-04-09T21:48:02.823 DEBUG:boto:Token: None 2024-04-09T21:48:02.823 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj44 2024-04-09T21:48:02.823 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7ubf6OvWfibIb4HoKG3tqvneB+M= 2024-04-09T21:48:02.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ubf6OvWfibIb4HoKG3tqvneB+M='} 2024-04-09T21:48:02.828 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0b0786880a39dd8-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.828 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.828 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.828 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.828 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.828 DEBUG:boto:Data: 2024-04-09T21:48:02.829 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.829 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.829 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.829 DEBUG:boto:Params: {} 2024-04-09T21:48:02.829 DEBUG:boto:Token: None 2024-04-09T21:48:02.829 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.829 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e18a4c1976ee83b4-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.831 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:48:02.831 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:48:02.831 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.832 DEBUG:boto:Path: /xryqqk-11/obj45 2024-04-09T21:48:02.832 DEBUG:boto:Data: 2024-04-09T21:48:02.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.832 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.832 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.832 DEBUG:boto:Params: {} 2024-04-09T21:48:02.832 DEBUG:boto:Token: None 2024-04-09T21:48:02.832 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj45 2024-04-09T21:48:02.832 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:J1nuQkVEP4qhdgFfWSw0+StPQLo= 2024-04-09T21:48:02.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J1nuQkVEP4qhdgFfWSw0+StPQLo='} 2024-04-09T21:48:02.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ab2c12310821055-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.875 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.875 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.875 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.875 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.875 DEBUG:boto:Data: 2024-04-09T21:48:02.875 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.875 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.875 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.875 DEBUG:boto:Params: {} 2024-04-09T21:48:02.876 DEBUG:boto:Token: None 2024-04-09T21:48:02.876 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.876 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008ddc4e58bdca595-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.878 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:48:02.878 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:48:02.879 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.879 DEBUG:boto:Path: /xryqqk-11/obj46 2024-04-09T21:48:02.879 DEBUG:boto:Data: 2024-04-09T21:48:02.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.879 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.879 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.879 DEBUG:boto:Params: {} 2024-04-09T21:48:02.879 DEBUG:boto:Token: None 2024-04-09T21:48:02.879 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj46 2024-04-09T21:48:02.880 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cJzKjVa04F3GAP6GHz+SV2G9Y7s= 2024-04-09T21:48:02.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cJzKjVa04F3GAP6GHz+SV2G9Y7s='} 2024-04-09T21:48:02.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000098d7a9628a141c-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.884 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.884 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.884 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.884 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.884 DEBUG:boto:Data: 2024-04-09T21:48:02.884 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.884 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.884 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.884 DEBUG:boto:Params: {} 2024-04-09T21:48:02.884 DEBUG:boto:Token: None 2024-04-09T21:48:02.885 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.885 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.887 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088f5f998f5d5c941-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.887 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:48:02.887 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:48:02.887 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.887 DEBUG:boto:Path: /xryqqk-11/obj47 2024-04-09T21:48:02.887 DEBUG:boto:Data: 2024-04-09T21:48:02.887 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.887 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.887 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.887 DEBUG:boto:Params: {} 2024-04-09T21:48:02.888 DEBUG:boto:Token: None 2024-04-09T21:48:02.888 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj47 2024-04-09T21:48:02.888 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RSeMqafVY9cxdOj/dH9f4EDbxfE= 2024-04-09T21:48:02.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RSeMqafVY9cxdOj/dH9f4EDbxfE='} 2024-04-09T21:48:02.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054319cdfb99f55b5-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.931 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.931 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.931 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.931 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.931 DEBUG:boto:Data: 2024-04-09T21:48:02.931 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.931 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.931 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.932 DEBUG:boto:Params: {} 2024-04-09T21:48:02.932 DEBUG:boto:Token: None 2024-04-09T21:48:02.932 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.932 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086a1f29cc9f88c60-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.934 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:48:02.934 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:48:02.934 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.935 DEBUG:boto:Path: /xryqqk-11/obj48 2024-04-09T21:48:02.935 DEBUG:boto:Data: 2024-04-09T21:48:02.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.935 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.935 DEBUG:boto:Params: {} 2024-04-09T21:48:02.935 DEBUG:boto:Token: None 2024-04-09T21:48:02.935 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj48 2024-04-09T21:48:02.935 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yOq0IY99m820Xsachf2BJa26PLM= 2024-04-09T21:48:02.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yOq0IY99m820Xsachf2BJa26PLM='} 2024-04-09T21:48:02.939 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e6737721b6a0e1-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.939 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:48:02.939 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:48:02.939 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.939 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:48:02.939 DEBUG:boto:Data: 2024-04-09T21:48:02.940 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.940 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.940 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.940 DEBUG:boto:Params: {} 2024-04-09T21:48:02.940 DEBUG:boto:Token: None 2024-04-09T21:48:02.940 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/ 2024-04-09T21:48:02.940 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-09T21:48:02.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-09T21:48:02.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3cc64815086bbac-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.942 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:48:02.942 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:48:02.943 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.943 DEBUG:boto:Path: /xryqqk-11/obj49 2024-04-09T21:48:02.943 DEBUG:boto:Data: 2024-04-09T21:48:02.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.943 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:48:02.943 DEBUG:boto:Port: 8000 2024-04-09T21:48:02.943 DEBUG:boto:Params: {} 2024-04-09T21:48:02.943 DEBUG:boto:Token: None 2024-04-09T21:48:02.943 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-11/obj49 2024-04-09T21:48:02.944 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w40feBMpHqA2Q/11CAwEMx/OakY= 2024-04-09T21:48:02.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w40feBMpHqA2Q/11CAwEMx/OakY='} 2024-04-09T21:48:02.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000e341eb0c6c917d-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.947 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:02.947 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:02.947 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.948 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:02.948 DEBUG:boto:Data: 2024-04-09T21:48:02.948 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.948 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:02.948 DEBUG:boto:Port: 8001 2024-04-09T21:48:02.948 DEBUG:boto:Params: {} 2024-04-09T21:48:02.948 DEBUG:boto:Token: None 2024-04-09T21:48:02.948 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-12/ 2024-04-09T21:48:02.948 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q= 2024-04-09T21:48:02.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q=', 'Content-Length': '0'} 2024-04-09T21:48:02.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f4ea8e66d2c808b-006615b792-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.951 DEBUG:boto:path=/xryqqk-12/obj1 2024-04-09T21:48:02.951 DEBUG:boto:auth_path=/xryqqk-12/obj1 2024-04-09T21:48:02.951 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.951 DEBUG:boto:Path: /xryqqk-12/obj1 2024-04-09T21:48:02.951 DEBUG:boto:Data: 2024-04-09T21:48:02.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.951 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:02.952 DEBUG:boto:Port: 8001 2024-04-09T21:48:02.952 DEBUG:boto:Params: {} 2024-04-09T21:48:02.952 DEBUG:boto:Token: None 2024-04-09T21:48:02.952 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-12/obj1 2024-04-09T21:48:02.952 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uzsMPj6IScWCB0t+YwseC7lmK4o= 2024-04-09T21:48:02.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uzsMPj6IScWCB0t+YwseC7lmK4o='} 2024-04-09T21:48:02.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000876b00436612f475-006615b792-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.958 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:02.958 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:02.958 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.958 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:02.958 DEBUG:boto:Data: 2024-04-09T21:48:02.958 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.958 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:02.959 DEBUG:boto:Port: 8001 2024-04-09T21:48:02.959 DEBUG:boto:Params: {} 2024-04-09T21:48:02.959 DEBUG:boto:Token: None 2024-04-09T21:48:02.959 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-12/ 2024-04-09T21:48:02.959 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q= 2024-04-09T21:48:02.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q=', 'Content-Length': '0'} 2024-04-09T21:48:02.961 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f771ea19d1bcc977-006615b792-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.961 DEBUG:boto:path=/xryqqk-12/obj2 2024-04-09T21:48:02.961 DEBUG:boto:auth_path=/xryqqk-12/obj2 2024-04-09T21:48:02.961 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.961 DEBUG:boto:Path: /xryqqk-12/obj2 2024-04-09T21:48:02.962 DEBUG:boto:Data: 2024-04-09T21:48:02.962 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.962 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:02.962 DEBUG:boto:Port: 8001 2024-04-09T21:48:02.962 DEBUG:boto:Params: {} 2024-04-09T21:48:02.962 DEBUG:boto:Token: None 2024-04-09T21:48:02.963 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-12/obj2 2024-04-09T21:48:02.963 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:NWAQhbK+xilZ8EV57XL56PzimbQ= 2024-04-09T21:48:02.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NWAQhbK+xilZ8EV57XL56PzimbQ='} 2024-04-09T21:48:02.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ce298dd4db9e7ce-006615b792-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.968 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:02.968 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:02.968 DEBUG:boto:Method: HEAD 2024-04-09T21:48:02.968 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:02.968 DEBUG:boto:Data: 2024-04-09T21:48:02.968 DEBUG:boto:Headers: {} 2024-04-09T21:48:02.968 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:02.969 DEBUG:boto:Port: 8001 2024-04-09T21:48:02.969 DEBUG:boto:Params: {} 2024-04-09T21:48:02.969 DEBUG:boto:Token: None 2024-04-09T21:48:02.969 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-12/ 2024-04-09T21:48:02.969 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q= 2024-04-09T21:48:02.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q=', 'Content-Length': '0'} 2024-04-09T21:48:02.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001991d7cd0597ec6-006615b792-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:02.971 DEBUG:boto:path=/xryqqk-12/obj3 2024-04-09T21:48:02.971 DEBUG:boto:auth_path=/xryqqk-12/obj3 2024-04-09T21:48:02.971 DEBUG:boto:Method: PUT 2024-04-09T21:48:02.971 DEBUG:boto:Path: /xryqqk-12/obj3 2024-04-09T21:48:02.971 DEBUG:boto:Data: 2024-04-09T21:48:02.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:02.972 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:02.972 DEBUG:boto:Port: 8001 2024-04-09T21:48:02.972 DEBUG:boto:Params: {} 2024-04-09T21:48:02.972 DEBUG:boto:Token: None 2024-04-09T21:48:02.972 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:02 GMT /xryqqk-12/obj3 2024-04-09T21:48:02.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8YqyamScLXaPibTjVDkCsmY/qq0= 2024-04-09T21:48:02.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8YqyamScLXaPibTjVDkCsmY/qq0='} 2024-04-09T21:48:03.014 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004acc6c833a9fba2f-006615b792-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.014 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.015 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.015 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.015 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.015 DEBUG:boto:Data: 2024-04-09T21:48:03.015 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.015 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.015 DEBUG:boto:Params: {} 2024-04-09T21:48:03.015 DEBUG:boto:Token: None 2024-04-09T21:48:03.016 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.016 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac7dc78e397c172a-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.018 DEBUG:boto:path=/xryqqk-12/obj4 2024-04-09T21:48:03.018 DEBUG:boto:auth_path=/xryqqk-12/obj4 2024-04-09T21:48:03.018 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.018 DEBUG:boto:Path: /xryqqk-12/obj4 2024-04-09T21:48:03.019 DEBUG:boto:Data: 2024-04-09T21:48:03.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.019 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.019 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.019 DEBUG:boto:Params: {} 2024-04-09T21:48:03.019 DEBUG:boto:Token: None 2024-04-09T21:48:03.019 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj4 2024-04-09T21:48:03.019 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SvAjPt495vAKiKkeoicsE8O4iZU= 2024-04-09T21:48:03.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SvAjPt495vAKiKkeoicsE8O4iZU='} 2024-04-09T21:48:03.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d66bcb5d243ccc4f-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.025 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.025 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.025 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.025 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.025 DEBUG:boto:Data: 2024-04-09T21:48:03.025 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.025 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.025 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.025 DEBUG:boto:Params: {} 2024-04-09T21:48:03.025 DEBUG:boto:Token: None 2024-04-09T21:48:03.025 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.026 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f3213a4a4877e58-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.028 DEBUG:boto:path=/xryqqk-12/obj5 2024-04-09T21:48:03.028 DEBUG:boto:auth_path=/xryqqk-12/obj5 2024-04-09T21:48:03.028 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.028 DEBUG:boto:Path: /xryqqk-12/obj5 2024-04-09T21:48:03.028 DEBUG:boto:Data: 2024-04-09T21:48:03.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.029 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.029 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.029 DEBUG:boto:Params: {} 2024-04-09T21:48:03.029 DEBUG:boto:Token: None 2024-04-09T21:48:03.029 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj5 2024-04-09T21:48:03.029 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y8pMymnTsYDpmJRDGTpCVuaMnY4= 2024-04-09T21:48:03.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y8pMymnTsYDpmJRDGTpCVuaMnY4='} 2024-04-09T21:48:03.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba5523ccedf10c93-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.033 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.033 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.033 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.033 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.033 DEBUG:boto:Data: 2024-04-09T21:48:03.033 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.033 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.033 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.033 DEBUG:boto:Params: {} 2024-04-09T21:48:03.033 DEBUG:boto:Token: None 2024-04-09T21:48:03.033 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.034 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.036 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afeee7711b807b50-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.036 DEBUG:boto:path=/xryqqk-12/obj6 2024-04-09T21:48:03.036 DEBUG:boto:auth_path=/xryqqk-12/obj6 2024-04-09T21:48:03.036 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.036 DEBUG:boto:Path: /xryqqk-12/obj6 2024-04-09T21:48:03.036 DEBUG:boto:Data: 2024-04-09T21:48:03.036 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.037 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.037 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.037 DEBUG:boto:Params: {} 2024-04-09T21:48:03.037 DEBUG:boto:Token: None 2024-04-09T21:48:03.037 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj6 2024-04-09T21:48:03.037 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kPxmNw3aa4zbKhaPLZ2as9IDig0= 2024-04-09T21:48:03.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kPxmNw3aa4zbKhaPLZ2as9IDig0='} 2024-04-09T21:48:03.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e535f02d5b823140-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.083 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.083 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.083 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.083 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.083 DEBUG:boto:Data: 2024-04-09T21:48:03.083 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.083 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.083 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.083 DEBUG:boto:Params: {} 2024-04-09T21:48:03.084 DEBUG:boto:Token: None 2024-04-09T21:48:03.084 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.084 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030feb54700647113-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.086 DEBUG:boto:path=/xryqqk-12/obj7 2024-04-09T21:48:03.087 DEBUG:boto:auth_path=/xryqqk-12/obj7 2024-04-09T21:48:03.087 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.087 DEBUG:boto:Path: /xryqqk-12/obj7 2024-04-09T21:48:03.087 DEBUG:boto:Data: 2024-04-09T21:48:03.087 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.087 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.087 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.087 DEBUG:boto:Params: {} 2024-04-09T21:48:03.087 DEBUG:boto:Token: None 2024-04-09T21:48:03.087 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj7 2024-04-09T21:48:03.087 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tPg7fGt8AesdSSpWKkKdqIcWRjg= 2024-04-09T21:48:03.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tPg7fGt8AesdSSpWKkKdqIcWRjg='} 2024-04-09T21:48:03.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e614e44132dafbed-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.091 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.091 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.091 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.091 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.091 DEBUG:boto:Data: 2024-04-09T21:48:03.091 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.091 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.092 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.092 DEBUG:boto:Params: {} 2024-04-09T21:48:03.092 DEBUG:boto:Token: None 2024-04-09T21:48:03.092 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.092 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.094 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000568ae60217dcef2-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.094 DEBUG:boto:path=/xryqqk-12/obj8 2024-04-09T21:48:03.095 DEBUG:boto:auth_path=/xryqqk-12/obj8 2024-04-09T21:48:03.095 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.095 DEBUG:boto:Path: /xryqqk-12/obj8 2024-04-09T21:48:03.095 DEBUG:boto:Data: 2024-04-09T21:48:03.095 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.095 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.095 DEBUG:boto:Params: {} 2024-04-09T21:48:03.095 DEBUG:boto:Token: None 2024-04-09T21:48:03.095 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj8 2024-04-09T21:48:03.096 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+g6BNPvY+nlblajGcQOdUYzvGGs= 2024-04-09T21:48:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+g6BNPvY+nlblajGcQOdUYzvGGs='} 2024-04-09T21:48:03.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095813fb8dafa7b5f-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.099 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.099 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.099 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.099 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.099 DEBUG:boto:Data: 2024-04-09T21:48:03.099 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.100 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.100 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.100 DEBUG:boto:Params: {} 2024-04-09T21:48:03.100 DEBUG:boto:Token: None 2024-04-09T21:48:03.100 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.100 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eec9b9013ffec45b-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.102 DEBUG:boto:path=/xryqqk-12/obj9 2024-04-09T21:48:03.102 DEBUG:boto:auth_path=/xryqqk-12/obj9 2024-04-09T21:48:03.103 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.103 DEBUG:boto:Path: /xryqqk-12/obj9 2024-04-09T21:48:03.103 DEBUG:boto:Data: 2024-04-09T21:48:03.103 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.103 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.103 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.103 DEBUG:boto:Params: {} 2024-04-09T21:48:03.103 DEBUG:boto:Token: None 2024-04-09T21:48:03.103 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj9 2024-04-09T21:48:03.103 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c7J+PFQGdMBezT1F/ex4ZcWYYfE= 2024-04-09T21:48:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c7J+PFQGdMBezT1F/ex4ZcWYYfE='} 2024-04-09T21:48:03.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e2f3efd86e4a072-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.147 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.147 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.147 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.147 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.147 DEBUG:boto:Data: 2024-04-09T21:48:03.147 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.147 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.147 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.147 DEBUG:boto:Params: {} 2024-04-09T21:48:03.148 DEBUG:boto:Token: None 2024-04-09T21:48:03.148 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.148 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031ba164699313a2e-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.150 DEBUG:boto:path=/xryqqk-12/obj10 2024-04-09T21:48:03.150 DEBUG:boto:auth_path=/xryqqk-12/obj10 2024-04-09T21:48:03.150 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.151 DEBUG:boto:Path: /xryqqk-12/obj10 2024-04-09T21:48:03.151 DEBUG:boto:Data: 2024-04-09T21:48:03.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.151 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.151 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.151 DEBUG:boto:Params: {} 2024-04-09T21:48:03.151 DEBUG:boto:Token: None 2024-04-09T21:48:03.151 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj10 2024-04-09T21:48:03.152 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uwkUcDjPftgAmWk6bT1daH43lqs= 2024-04-09T21:48:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uwkUcDjPftgAmWk6bT1daH43lqs='} 2024-04-09T21:48:03.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009c0e787083d4dad-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.162 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.162 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.162 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.162 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.163 DEBUG:boto:Data: 2024-04-09T21:48:03.163 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.163 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.163 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.163 DEBUG:boto:Params: {} 2024-04-09T21:48:03.163 DEBUG:boto:Token: None 2024-04-09T21:48:03.163 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.163 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a06582044ef7dd5-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.165 DEBUG:boto:path=/xryqqk-12/obj11 2024-04-09T21:48:03.166 DEBUG:boto:auth_path=/xryqqk-12/obj11 2024-04-09T21:48:03.166 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.166 DEBUG:boto:Path: /xryqqk-12/obj11 2024-04-09T21:48:03.166 DEBUG:boto:Data: 2024-04-09T21:48:03.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.166 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.166 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.166 DEBUG:boto:Params: {} 2024-04-09T21:48:03.166 DEBUG:boto:Token: None 2024-04-09T21:48:03.167 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj11 2024-04-09T21:48:03.167 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PVheb51mJkOSvxS6QA+Qy2mdG5I= 2024-04-09T21:48:03.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PVheb51mJkOSvxS6QA+Qy2mdG5I='} 2024-04-09T21:48:03.172 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022508b49228f4498-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.172 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.172 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.172 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.172 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.172 DEBUG:boto:Data: 2024-04-09T21:48:03.172 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.172 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.172 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.173 DEBUG:boto:Params: {} 2024-04-09T21:48:03.173 DEBUG:boto:Token: None 2024-04-09T21:48:03.173 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.173 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a33cf1325f0d0314-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.176 DEBUG:boto:path=/xryqqk-12/obj12 2024-04-09T21:48:03.176 DEBUG:boto:auth_path=/xryqqk-12/obj12 2024-04-09T21:48:03.176 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.176 DEBUG:boto:Path: /xryqqk-12/obj12 2024-04-09T21:48:03.176 DEBUG:boto:Data: 2024-04-09T21:48:03.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.176 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.176 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.176 DEBUG:boto:Params: {} 2024-04-09T21:48:03.177 DEBUG:boto:Token: None 2024-04-09T21:48:03.177 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj12 2024-04-09T21:48:03.177 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wg6bp56vHI0k/IMQH6Y1Ae9OUz4= 2024-04-09T21:48:03.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wg6bp56vHI0k/IMQH6Y1Ae9OUz4='} 2024-04-09T21:48:03.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f91ce26dca546f5-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.219 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.219 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.219 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.219 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.219 DEBUG:boto:Data: 2024-04-09T21:48:03.219 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.219 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.219 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.219 DEBUG:boto:Params: {} 2024-04-09T21:48:03.220 DEBUG:boto:Token: None 2024-04-09T21:48:03.220 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.220 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecca3b6beee1363d-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.222 DEBUG:boto:path=/xryqqk-12/obj13 2024-04-09T21:48:03.223 DEBUG:boto:auth_path=/xryqqk-12/obj13 2024-04-09T21:48:03.223 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.223 DEBUG:boto:Path: /xryqqk-12/obj13 2024-04-09T21:48:03.223 DEBUG:boto:Data: 2024-04-09T21:48:03.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.223 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.223 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.223 DEBUG:boto:Params: {} 2024-04-09T21:48:03.223 DEBUG:boto:Token: None 2024-04-09T21:48:03.223 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj13 2024-04-09T21:48:03.223 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:juui1wrwD/LiPlnrjZsN+EMxZJ0= 2024-04-09T21:48:03.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:juui1wrwD/LiPlnrjZsN+EMxZJ0='} 2024-04-09T21:48:03.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d693f823bec30fb-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.227 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.227 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.227 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.227 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.227 DEBUG:boto:Data: 2024-04-09T21:48:03.227 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.228 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.228 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.228 DEBUG:boto:Params: {} 2024-04-09T21:48:03.228 DEBUG:boto:Token: None 2024-04-09T21:48:03.228 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.228 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001192c082aceec0e5-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.230 DEBUG:boto:path=/xryqqk-12/obj14 2024-04-09T21:48:03.231 DEBUG:boto:auth_path=/xryqqk-12/obj14 2024-04-09T21:48:03.231 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.231 DEBUG:boto:Path: /xryqqk-12/obj14 2024-04-09T21:48:03.231 DEBUG:boto:Data: 2024-04-09T21:48:03.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.231 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.231 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.231 DEBUG:boto:Params: {} 2024-04-09T21:48:03.231 DEBUG:boto:Token: None 2024-04-09T21:48:03.231 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj14 2024-04-09T21:48:03.231 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CLKH09adS73uWnHrQruxC8rQP+Q= 2024-04-09T21:48:03.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CLKH09adS73uWnHrQruxC8rQP+Q='} 2024-04-09T21:48:03.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cc2e0e9896cdf25-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.235 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.235 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.235 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.235 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.236 DEBUG:boto:Data: 2024-04-09T21:48:03.236 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.236 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.236 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.236 DEBUG:boto:Params: {} 2024-04-09T21:48:03.236 DEBUG:boto:Token: None 2024-04-09T21:48:03.236 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.236 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8edc2767f409ad0-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.239 DEBUG:boto:path=/xryqqk-12/obj15 2024-04-09T21:48:03.239 DEBUG:boto:auth_path=/xryqqk-12/obj15 2024-04-09T21:48:03.239 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.239 DEBUG:boto:Path: /xryqqk-12/obj15 2024-04-09T21:48:03.239 DEBUG:boto:Data: 2024-04-09T21:48:03.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.239 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.239 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.239 DEBUG:boto:Params: {} 2024-04-09T21:48:03.239 DEBUG:boto:Token: None 2024-04-09T21:48:03.240 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj15 2024-04-09T21:48:03.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nud1lICN0zogbFf1xeaFv4Gtxqo= 2024-04-09T21:48:03.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nud1lICN0zogbFf1xeaFv4Gtxqo='} 2024-04-09T21:48:03.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8e6366840670526-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.283 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.283 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.283 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.283 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.283 DEBUG:boto:Data: 2024-04-09T21:48:03.283 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.284 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.284 DEBUG:boto:Params: {} 2024-04-09T21:48:03.284 DEBUG:boto:Token: None 2024-04-09T21:48:03.284 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.284 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea25a09c7014450c-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.287 DEBUG:boto:path=/xryqqk-12/obj16 2024-04-09T21:48:03.287 DEBUG:boto:auth_path=/xryqqk-12/obj16 2024-04-09T21:48:03.287 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.287 DEBUG:boto:Path: /xryqqk-12/obj16 2024-04-09T21:48:03.287 DEBUG:boto:Data: 2024-04-09T21:48:03.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.287 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.287 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.287 DEBUG:boto:Params: {} 2024-04-09T21:48:03.287 DEBUG:boto:Token: None 2024-04-09T21:48:03.288 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj16 2024-04-09T21:48:03.288 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XhKUyj906dtFrv3xBtx/joY/14E= 2024-04-09T21:48:03.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XhKUyj906dtFrv3xBtx/joY/14E='} 2024-04-09T21:48:03.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000428895689ff0f48c-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.294 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.294 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.294 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.295 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.295 DEBUG:boto:Data: 2024-04-09T21:48:03.295 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.295 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.295 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.295 DEBUG:boto:Params: {} 2024-04-09T21:48:03.295 DEBUG:boto:Token: None 2024-04-09T21:48:03.295 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.296 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.298 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5e760d5db72fc2b-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.298 DEBUG:boto:path=/xryqqk-12/obj17 2024-04-09T21:48:03.298 DEBUG:boto:auth_path=/xryqqk-12/obj17 2024-04-09T21:48:03.298 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.298 DEBUG:boto:Path: /xryqqk-12/obj17 2024-04-09T21:48:03.298 DEBUG:boto:Data: 2024-04-09T21:48:03.299 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.299 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.299 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.299 DEBUG:boto:Params: {} 2024-04-09T21:48:03.299 DEBUG:boto:Token: None 2024-04-09T21:48:03.299 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj17 2024-04-09T21:48:03.300 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uq/+RAETIfBdXtZ6RAIBvo7M+/k= 2024-04-09T21:48:03.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uq/+RAETIfBdXtZ6RAIBvo7M+/k='} 2024-04-09T21:48:03.343 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000eb0b6e9acaeacb6-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.343 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.343 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.343 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.343 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.343 DEBUG:boto:Data: 2024-04-09T21:48:03.343 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.343 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.344 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.344 DEBUG:boto:Params: {} 2024-04-09T21:48:03.344 DEBUG:boto:Token: None 2024-04-09T21:48:03.344 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.344 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000469cb12e97f7105-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.347 DEBUG:boto:path=/xryqqk-12/obj18 2024-04-09T21:48:03.347 DEBUG:boto:auth_path=/xryqqk-12/obj18 2024-04-09T21:48:03.347 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.347 DEBUG:boto:Path: /xryqqk-12/obj18 2024-04-09T21:48:03.347 DEBUG:boto:Data: 2024-04-09T21:48:03.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.347 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.347 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.347 DEBUG:boto:Params: {} 2024-04-09T21:48:03.347 DEBUG:boto:Token: None 2024-04-09T21:48:03.347 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj18 2024-04-09T21:48:03.348 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cStLKI83yRHN72/QXnt/tXqfv04= 2024-04-09T21:48:03.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cStLKI83yRHN72/QXnt/tXqfv04='} 2024-04-09T21:48:03.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cc7ee4f20bef73f-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.351 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.351 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.352 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.352 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.352 DEBUG:boto:Data: 2024-04-09T21:48:03.352 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.352 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.352 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.352 DEBUG:boto:Params: {} 2024-04-09T21:48:03.352 DEBUG:boto:Token: None 2024-04-09T21:48:03.352 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.352 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2cd716ab06efc77-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.354 DEBUG:boto:path=/xryqqk-12/obj19 2024-04-09T21:48:03.355 DEBUG:boto:auth_path=/xryqqk-12/obj19 2024-04-09T21:48:03.355 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.355 DEBUG:boto:Path: /xryqqk-12/obj19 2024-04-09T21:48:03.355 DEBUG:boto:Data: 2024-04-09T21:48:03.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.355 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.355 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.355 DEBUG:boto:Params: {} 2024-04-09T21:48:03.355 DEBUG:boto:Token: None 2024-04-09T21:48:03.355 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj19 2024-04-09T21:48:03.355 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:BTVEnPHfiAnSlODyz5xWBFzjb5k= 2024-04-09T21:48:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BTVEnPHfiAnSlODyz5xWBFzjb5k='} 2024-04-09T21:48:03.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ae012926e48c040-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.399 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.399 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.399 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.399 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.399 DEBUG:boto:Data: 2024-04-09T21:48:03.399 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.400 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.400 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.400 DEBUG:boto:Params: {} 2024-04-09T21:48:03.400 DEBUG:boto:Token: None 2024-04-09T21:48:03.400 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.400 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008addd01b259e52c9-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.403 DEBUG:boto:path=/xryqqk-12/obj20 2024-04-09T21:48:03.403 DEBUG:boto:auth_path=/xryqqk-12/obj20 2024-04-09T21:48:03.403 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.403 DEBUG:boto:Path: /xryqqk-12/obj20 2024-04-09T21:48:03.403 DEBUG:boto:Data: 2024-04-09T21:48:03.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.403 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.404 DEBUG:boto:Params: {} 2024-04-09T21:48:03.404 DEBUG:boto:Token: None 2024-04-09T21:48:03.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj20 2024-04-09T21:48:03.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:T3l7YQEedu1Y98RZjb2GgsGRiYg= 2024-04-09T21:48:03.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T3l7YQEedu1Y98RZjb2GgsGRiYg='} 2024-04-09T21:48:03.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd961b341d5390ed-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.409 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.409 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.409 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.409 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.410 DEBUG:boto:Data: 2024-04-09T21:48:03.410 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.410 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.410 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.410 DEBUG:boto:Params: {} 2024-04-09T21:48:03.410 DEBUG:boto:Token: None 2024-04-09T21:48:03.410 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.411 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000257b41e9e85c33bc-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.413 DEBUG:boto:path=/xryqqk-12/obj21 2024-04-09T21:48:03.413 DEBUG:boto:auth_path=/xryqqk-12/obj21 2024-04-09T21:48:03.413 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.413 DEBUG:boto:Path: /xryqqk-12/obj21 2024-04-09T21:48:03.413 DEBUG:boto:Data: 2024-04-09T21:48:03.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.414 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.414 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.414 DEBUG:boto:Params: {} 2024-04-09T21:48:03.414 DEBUG:boto:Token: None 2024-04-09T21:48:03.414 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj21 2024-04-09T21:48:03.414 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uVcPnDl5XE6S1X0PymqtqWFyrhg= 2024-04-09T21:48:03.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uVcPnDl5XE6S1X0PymqtqWFyrhg='} 2024-04-09T21:48:03.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002391d32d6737b9c9-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.463 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.463 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.463 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.464 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.464 DEBUG:boto:Data: 2024-04-09T21:48:03.464 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.464 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.464 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.464 DEBUG:boto:Params: {} 2024-04-09T21:48:03.464 DEBUG:boto:Token: None 2024-04-09T21:48:03.464 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020ee9ad31e335f11-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.467 DEBUG:boto:path=/xryqqk-12/obj22 2024-04-09T21:48:03.467 DEBUG:boto:auth_path=/xryqqk-12/obj22 2024-04-09T21:48:03.467 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.467 DEBUG:boto:Path: /xryqqk-12/obj22 2024-04-09T21:48:03.467 DEBUG:boto:Data: 2024-04-09T21:48:03.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.467 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.468 DEBUG:boto:Params: {} 2024-04-09T21:48:03.468 DEBUG:boto:Token: None 2024-04-09T21:48:03.468 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj22 2024-04-09T21:48:03.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Jn/dP+IrA4rTNp1oVZZZr43H5Xc= 2024-04-09T21:48:03.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Jn/dP+IrA4rTNp1oVZZZr43H5Xc='} 2024-04-09T21:48:03.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aae320511f3386e3-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.473 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.473 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.474 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.474 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.474 DEBUG:boto:Data: 2024-04-09T21:48:03.474 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.474 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.474 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.474 DEBUG:boto:Params: {} 2024-04-09T21:48:03.474 DEBUG:boto:Token: None 2024-04-09T21:48:03.474 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.475 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbec4fb9bf1e07b7-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.477 DEBUG:boto:path=/xryqqk-12/obj23 2024-04-09T21:48:03.477 DEBUG:boto:auth_path=/xryqqk-12/obj23 2024-04-09T21:48:03.478 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.478 DEBUG:boto:Path: /xryqqk-12/obj23 2024-04-09T21:48:03.478 DEBUG:boto:Data: 2024-04-09T21:48:03.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.478 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.478 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.478 DEBUG:boto:Params: {} 2024-04-09T21:48:03.478 DEBUG:boto:Token: None 2024-04-09T21:48:03.478 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj23 2024-04-09T21:48:03.479 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:J5Ih/Af4Rywj7wt24/jh8R2B+EA= 2024-04-09T21:48:03.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J5Ih/Af4Rywj7wt24/jh8R2B+EA='} 2024-04-09T21:48:03.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cdfd5bac3149373-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.523 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.523 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.523 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.523 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.523 DEBUG:boto:Data: 2024-04-09T21:48:03.523 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.523 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.523 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.523 DEBUG:boto:Params: {} 2024-04-09T21:48:03.523 DEBUG:boto:Token: None 2024-04-09T21:48:03.524 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.524 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a42250d9814d60eb-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.526 DEBUG:boto:path=/xryqqk-12/obj24 2024-04-09T21:48:03.526 DEBUG:boto:auth_path=/xryqqk-12/obj24 2024-04-09T21:48:03.526 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.526 DEBUG:boto:Path: /xryqqk-12/obj24 2024-04-09T21:48:03.527 DEBUG:boto:Data: 2024-04-09T21:48:03.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.527 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.527 DEBUG:boto:Params: {} 2024-04-09T21:48:03.527 DEBUG:boto:Token: None 2024-04-09T21:48:03.527 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj24 2024-04-09T21:48:03.527 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HozymsBHn2VijBA6b/JEUyorRZw= 2024-04-09T21:48:03.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HozymsBHn2VijBA6b/JEUyorRZw='} 2024-04-09T21:48:03.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5c79178ab24c638-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.531 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.531 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.531 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.531 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.531 DEBUG:boto:Data: 2024-04-09T21:48:03.531 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.532 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.532 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.532 DEBUG:boto:Params: {} 2024-04-09T21:48:03.532 DEBUG:boto:Token: None 2024-04-09T21:48:03.532 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000786847deaaeda736-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.534 DEBUG:boto:path=/xryqqk-12/obj25 2024-04-09T21:48:03.535 DEBUG:boto:auth_path=/xryqqk-12/obj25 2024-04-09T21:48:03.535 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.535 DEBUG:boto:Path: /xryqqk-12/obj25 2024-04-09T21:48:03.535 DEBUG:boto:Data: 2024-04-09T21:48:03.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.535 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.535 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.535 DEBUG:boto:Params: {} 2024-04-09T21:48:03.535 DEBUG:boto:Token: None 2024-04-09T21:48:03.535 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj25 2024-04-09T21:48:03.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qG34/Gm3fs4/t5TVY5M2glOns2o= 2024-04-09T21:48:03.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qG34/Gm3fs4/t5TVY5M2glOns2o='} 2024-04-09T21:48:03.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052139afcc1fb9eeb-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.583 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.583 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.583 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.583 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.583 DEBUG:boto:Data: 2024-04-09T21:48:03.583 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.583 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.583 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.583 DEBUG:boto:Params: {} 2024-04-09T21:48:03.583 DEBUG:boto:Token: None 2024-04-09T21:48:03.583 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.584 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7b5972e06dd06ec-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.586 DEBUG:boto:path=/xryqqk-12/obj26 2024-04-09T21:48:03.586 DEBUG:boto:auth_path=/xryqqk-12/obj26 2024-04-09T21:48:03.586 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.586 DEBUG:boto:Path: /xryqqk-12/obj26 2024-04-09T21:48:03.587 DEBUG:boto:Data: 2024-04-09T21:48:03.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.587 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.587 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.587 DEBUG:boto:Params: {} 2024-04-09T21:48:03.587 DEBUG:boto:Token: None 2024-04-09T21:48:03.587 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj26 2024-04-09T21:48:03.587 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yPVqljTRRY7VS6uJ+/+e0gURjps= 2024-04-09T21:48:03.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yPVqljTRRY7VS6uJ+/+e0gURjps='} 2024-04-09T21:48:03.591 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4b333a86f3eeebf-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.591 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.591 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.591 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.591 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.591 DEBUG:boto:Data: 2024-04-09T21:48:03.591 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.591 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.591 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.591 DEBUG:boto:Params: {} 2024-04-09T21:48:03.591 DEBUG:boto:Token: None 2024-04-09T21:48:03.592 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b79a4ab406d67d8c-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.594 DEBUG:boto:path=/xryqqk-12/obj27 2024-04-09T21:48:03.594 DEBUG:boto:auth_path=/xryqqk-12/obj27 2024-04-09T21:48:03.594 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.594 DEBUG:boto:Path: /xryqqk-12/obj27 2024-04-09T21:48:03.594 DEBUG:boto:Data: 2024-04-09T21:48:03.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.594 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.595 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.595 DEBUG:boto:Params: {} 2024-04-09T21:48:03.595 DEBUG:boto:Token: None 2024-04-09T21:48:03.595 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj27 2024-04-09T21:48:03.595 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:di1nN7RnI8c4yI5Ndo2Z3IzDRUs= 2024-04-09T21:48:03.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:di1nN7RnI8c4yI5Ndo2Z3IzDRUs='} 2024-04-09T21:48:03.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f63b4dd205fac368-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.599 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.599 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.599 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.599 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.599 DEBUG:boto:Data: 2024-04-09T21:48:03.599 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.599 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.599 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.599 DEBUG:boto:Params: {} 2024-04-09T21:48:03.599 DEBUG:boto:Token: None 2024-04-09T21:48:03.599 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.600 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa29b0d3a58d6c05-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.602 DEBUG:boto:path=/xryqqk-12/obj28 2024-04-09T21:48:03.602 DEBUG:boto:auth_path=/xryqqk-12/obj28 2024-04-09T21:48:03.602 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.602 DEBUG:boto:Path: /xryqqk-12/obj28 2024-04-09T21:48:03.602 DEBUG:boto:Data: 2024-04-09T21:48:03.602 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.602 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.603 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.603 DEBUG:boto:Params: {} 2024-04-09T21:48:03.603 DEBUG:boto:Token: None 2024-04-09T21:48:03.603 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj28 2024-04-09T21:48:03.603 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLaqBniVM5u053a/tIjOuAeR/Jo= 2024-04-09T21:48:03.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLaqBniVM5u053a/tIjOuAeR/Jo='} 2024-04-09T21:48:03.646 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb1515e7fe516ddc-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.647 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.647 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.647 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.647 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.647 DEBUG:boto:Data: 2024-04-09T21:48:03.647 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.647 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.647 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.647 DEBUG:boto:Params: {} 2024-04-09T21:48:03.647 DEBUG:boto:Token: None 2024-04-09T21:48:03.648 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.648 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d69994fa1fa46ae0-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.650 DEBUG:boto:path=/xryqqk-12/obj29 2024-04-09T21:48:03.650 DEBUG:boto:auth_path=/xryqqk-12/obj29 2024-04-09T21:48:03.650 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.650 DEBUG:boto:Path: /xryqqk-12/obj29 2024-04-09T21:48:03.651 DEBUG:boto:Data: 2024-04-09T21:48:03.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.651 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.651 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.651 DEBUG:boto:Params: {} 2024-04-09T21:48:03.651 DEBUG:boto:Token: None 2024-04-09T21:48:03.651 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj29 2024-04-09T21:48:03.651 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Na8hESqpD+Lrv6vU0WH69V4vwxM= 2024-04-09T21:48:03.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Na8hESqpD+Lrv6vU0WH69V4vwxM='} 2024-04-09T21:48:03.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1adff8a0612d30b-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.655 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.655 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.655 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.655 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.655 DEBUG:boto:Data: 2024-04-09T21:48:03.655 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.655 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.655 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.655 DEBUG:boto:Params: {} 2024-04-09T21:48:03.656 DEBUG:boto:Token: None 2024-04-09T21:48:03.656 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fb11c118a911e6d-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.658 DEBUG:boto:path=/xryqqk-12/obj30 2024-04-09T21:48:03.658 DEBUG:boto:auth_path=/xryqqk-12/obj30 2024-04-09T21:48:03.658 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.658 DEBUG:boto:Path: /xryqqk-12/obj30 2024-04-09T21:48:03.658 DEBUG:boto:Data: 2024-04-09T21:48:03.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.659 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.659 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.659 DEBUG:boto:Params: {} 2024-04-09T21:48:03.659 DEBUG:boto:Token: None 2024-04-09T21:48:03.659 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj30 2024-04-09T21:48:03.659 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XVvaLWNZ7fFInV5KZeoUOyayyp8= 2024-04-09T21:48:03.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XVvaLWNZ7fFInV5KZeoUOyayyp8='} 2024-04-09T21:48:03.662 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff3572dd9f3ea616-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.663 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.663 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.663 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.663 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.663 DEBUG:boto:Data: 2024-04-09T21:48:03.663 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.663 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.663 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.663 DEBUG:boto:Params: {} 2024-04-09T21:48:03.664 DEBUG:boto:Token: None 2024-04-09T21:48:03.664 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.666 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dacb105c28158127-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.666 DEBUG:boto:path=/xryqqk-12/obj31 2024-04-09T21:48:03.666 DEBUG:boto:auth_path=/xryqqk-12/obj31 2024-04-09T21:48:03.666 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.667 DEBUG:boto:Path: /xryqqk-12/obj31 2024-04-09T21:48:03.667 DEBUG:boto:Data: 2024-04-09T21:48:03.667 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.667 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.667 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.667 DEBUG:boto:Params: {} 2024-04-09T21:48:03.667 DEBUG:boto:Token: None 2024-04-09T21:48:03.667 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj31 2024-04-09T21:48:03.667 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n5Z4xtG1SeNMzFoV3QxAkxBErAg= 2024-04-09T21:48:03.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n5Z4xtG1SeNMzFoV3QxAkxBErAg='} 2024-04-09T21:48:03.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6961af09730aa16-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.711 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.711 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.711 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.711 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.711 DEBUG:boto:Data: 2024-04-09T21:48:03.711 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.711 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.712 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.712 DEBUG:boto:Params: {} 2024-04-09T21:48:03.712 DEBUG:boto:Token: None 2024-04-09T21:48:03.712 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.712 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.714 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f24734ba78aa028-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.715 DEBUG:boto:path=/xryqqk-12/obj32 2024-04-09T21:48:03.715 DEBUG:boto:auth_path=/xryqqk-12/obj32 2024-04-09T21:48:03.715 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.715 DEBUG:boto:Path: /xryqqk-12/obj32 2024-04-09T21:48:03.715 DEBUG:boto:Data: 2024-04-09T21:48:03.715 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.715 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.715 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.715 DEBUG:boto:Params: {} 2024-04-09T21:48:03.716 DEBUG:boto:Token: None 2024-04-09T21:48:03.716 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj32 2024-04-09T21:48:03.716 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6qvAlh+lTUs1Ikz3ch2MnHHF8RI= 2024-04-09T21:48:03.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6qvAlh+lTUs1Ikz3ch2MnHHF8RI='} 2024-04-09T21:48:03.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf71df5c1c39b7cd-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.720 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.720 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.720 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.720 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.720 DEBUG:boto:Data: 2024-04-09T21:48:03.720 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.720 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.720 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.720 DEBUG:boto:Params: {} 2024-04-09T21:48:03.721 DEBUG:boto:Token: None 2024-04-09T21:48:03.721 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.721 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8d6be22f1623d9d-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.723 DEBUG:boto:path=/xryqqk-12/obj33 2024-04-09T21:48:03.723 DEBUG:boto:auth_path=/xryqqk-12/obj33 2024-04-09T21:48:03.723 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.723 DEBUG:boto:Path: /xryqqk-12/obj33 2024-04-09T21:48:03.723 DEBUG:boto:Data: 2024-04-09T21:48:03.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.723 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.723 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.724 DEBUG:boto:Params: {} 2024-04-09T21:48:03.724 DEBUG:boto:Token: None 2024-04-09T21:48:03.724 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj33 2024-04-09T21:48:03.724 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hlJwxawneth5PJApWfevcfiVZ1E= 2024-04-09T21:48:03.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hlJwxawneth5PJApWfevcfiVZ1E='} 2024-04-09T21:48:03.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007928afb4d7ae2dd7-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.767 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.767 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.767 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.767 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.767 DEBUG:boto:Data: 2024-04-09T21:48:03.768 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.768 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.768 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.768 DEBUG:boto:Params: {} 2024-04-09T21:48:03.768 DEBUG:boto:Token: None 2024-04-09T21:48:03.768 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.768 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004fe107373dc22a8-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.771 DEBUG:boto:path=/xryqqk-12/obj34 2024-04-09T21:48:03.771 DEBUG:boto:auth_path=/xryqqk-12/obj34 2024-04-09T21:48:03.771 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.771 DEBUG:boto:Path: /xryqqk-12/obj34 2024-04-09T21:48:03.771 DEBUG:boto:Data: 2024-04-09T21:48:03.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.771 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.771 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.772 DEBUG:boto:Params: {} 2024-04-09T21:48:03.772 DEBUG:boto:Token: None 2024-04-09T21:48:03.772 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj34 2024-04-09T21:48:03.772 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:I+gBbHDNdINxzgKTZ+C7OZuu4ZM= 2024-04-09T21:48:03.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I+gBbHDNdINxzgKTZ+C7OZuu4ZM='} 2024-04-09T21:48:03.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f732ec28839af47-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.775 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.776 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.776 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.776 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.776 DEBUG:boto:Data: 2024-04-09T21:48:03.776 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.776 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.776 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.776 DEBUG:boto:Params: {} 2024-04-09T21:48:03.776 DEBUG:boto:Token: None 2024-04-09T21:48:03.776 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.777 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fa1285f5c3f4eab-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.779 DEBUG:boto:path=/xryqqk-12/obj35 2024-04-09T21:48:03.779 DEBUG:boto:auth_path=/xryqqk-12/obj35 2024-04-09T21:48:03.779 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.779 DEBUG:boto:Path: /xryqqk-12/obj35 2024-04-09T21:48:03.779 DEBUG:boto:Data: 2024-04-09T21:48:03.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.780 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.780 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.780 DEBUG:boto:Params: {} 2024-04-09T21:48:03.780 DEBUG:boto:Token: None 2024-04-09T21:48:03.780 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj35 2024-04-09T21:48:03.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Kt7/kGsSwacCHEzvqKVMupO7JXw= 2024-04-09T21:48:03.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kt7/kGsSwacCHEzvqKVMupO7JXw='} 2024-04-09T21:48:03.822 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000746d0639f1b53a6e-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.823 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.823 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.823 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.823 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.823 DEBUG:boto:Data: 2024-04-09T21:48:03.823 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.823 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.824 DEBUG:boto:Params: {} 2024-04-09T21:48:03.824 DEBUG:boto:Token: None 2024-04-09T21:48:03.824 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c05e198d4249df90-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.826 DEBUG:boto:path=/xryqqk-12/obj36 2024-04-09T21:48:03.826 DEBUG:boto:auth_path=/xryqqk-12/obj36 2024-04-09T21:48:03.827 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.827 DEBUG:boto:Path: /xryqqk-12/obj36 2024-04-09T21:48:03.827 DEBUG:boto:Data: 2024-04-09T21:48:03.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.827 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.827 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.827 DEBUG:boto:Params: {} 2024-04-09T21:48:03.827 DEBUG:boto:Token: None 2024-04-09T21:48:03.827 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj36 2024-04-09T21:48:03.828 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mtbcm+g/xHMJytbPNqIq9YgAl3o= 2024-04-09T21:48:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mtbcm+g/xHMJytbPNqIq9YgAl3o='} 2024-04-09T21:48:03.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c51198a1f8345f75-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.831 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.832 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.832 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.832 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.832 DEBUG:boto:Data: 2024-04-09T21:48:03.832 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.832 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.832 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.832 DEBUG:boto:Params: {} 2024-04-09T21:48:03.832 DEBUG:boto:Token: None 2024-04-09T21:48:03.832 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.833 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be198d5886b9f543-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.835 DEBUG:boto:path=/xryqqk-12/obj37 2024-04-09T21:48:03.835 DEBUG:boto:auth_path=/xryqqk-12/obj37 2024-04-09T21:48:03.835 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.835 DEBUG:boto:Path: /xryqqk-12/obj37 2024-04-09T21:48:03.835 DEBUG:boto:Data: 2024-04-09T21:48:03.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.836 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.836 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.836 DEBUG:boto:Params: {} 2024-04-09T21:48:03.836 DEBUG:boto:Token: None 2024-04-09T21:48:03.836 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj37 2024-04-09T21:48:03.836 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:g3TF7lG1CbtnuFX3nXtMqwWw8Zo= 2024-04-09T21:48:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:g3TF7lG1CbtnuFX3nXtMqwWw8Zo='} 2024-04-09T21:48:03.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aff201b8b2d8aa7e-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.879 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.879 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.879 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.879 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.879 DEBUG:boto:Data: 2024-04-09T21:48:03.879 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.879 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.879 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.879 DEBUG:boto:Params: {} 2024-04-09T21:48:03.880 DEBUG:boto:Token: None 2024-04-09T21:48:03.880 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.880 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba75d984b462bbe5-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.882 DEBUG:boto:path=/xryqqk-12/obj38 2024-04-09T21:48:03.883 DEBUG:boto:auth_path=/xryqqk-12/obj38 2024-04-09T21:48:03.883 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.883 DEBUG:boto:Path: /xryqqk-12/obj38 2024-04-09T21:48:03.883 DEBUG:boto:Data: 2024-04-09T21:48:03.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.883 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.883 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.883 DEBUG:boto:Params: {} 2024-04-09T21:48:03.883 DEBUG:boto:Token: None 2024-04-09T21:48:03.883 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj38 2024-04-09T21:48:03.884 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+QyNQvcXQzXbZAC0XyoPdMyycUw= 2024-04-09T21:48:03.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+QyNQvcXQzXbZAC0XyoPdMyycUw='} 2024-04-09T21:48:03.887 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a071bc8a145dcfa7-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.887 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.888 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.888 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.888 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.888 DEBUG:boto:Data: 2024-04-09T21:48:03.888 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.888 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.888 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.888 DEBUG:boto:Params: {} 2024-04-09T21:48:03.888 DEBUG:boto:Token: None 2024-04-09T21:48:03.888 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.889 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f43fbacd3706f493-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.891 DEBUG:boto:path=/xryqqk-12/obj39 2024-04-09T21:48:03.891 DEBUG:boto:auth_path=/xryqqk-12/obj39 2024-04-09T21:48:03.891 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.891 DEBUG:boto:Path: /xryqqk-12/obj39 2024-04-09T21:48:03.891 DEBUG:boto:Data: 2024-04-09T21:48:03.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.892 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.892 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.892 DEBUG:boto:Params: {} 2024-04-09T21:48:03.892 DEBUG:boto:Token: None 2024-04-09T21:48:03.892 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj39 2024-04-09T21:48:03.892 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vYRXWpsWm/mH2tRVPaxQSTwT328= 2024-04-09T21:48:03.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vYRXWpsWm/mH2tRVPaxQSTwT328='} 2024-04-09T21:48:03.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ff8541030cedbd7-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.935 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.935 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.935 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.935 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.935 DEBUG:boto:Data: 2024-04-09T21:48:03.935 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.935 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.935 DEBUG:boto:Params: {} 2024-04-09T21:48:03.936 DEBUG:boto:Token: None 2024-04-09T21:48:03.936 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000986a9aa99bf7d960-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.938 DEBUG:boto:path=/xryqqk-12/obj40 2024-04-09T21:48:03.938 DEBUG:boto:auth_path=/xryqqk-12/obj40 2024-04-09T21:48:03.939 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.939 DEBUG:boto:Path: /xryqqk-12/obj40 2024-04-09T21:48:03.939 DEBUG:boto:Data: 2024-04-09T21:48:03.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.939 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.939 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.939 DEBUG:boto:Params: {} 2024-04-09T21:48:03.939 DEBUG:boto:Token: None 2024-04-09T21:48:03.939 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj40 2024-04-09T21:48:03.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q1Iw26zfxaefIIj6oncSmG+QmzA= 2024-04-09T21:48:03.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q1Iw26zfxaefIIj6oncSmG+QmzA='} 2024-04-09T21:48:03.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e647ba874532be5d-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.943 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.943 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.943 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.943 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.943 DEBUG:boto:Data: 2024-04-09T21:48:03.943 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.944 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.944 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.944 DEBUG:boto:Params: {} 2024-04-09T21:48:03.944 DEBUG:boto:Token: None 2024-04-09T21:48:03.944 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.944 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.946 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f1bf3ac8bc7c1a2-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.946 DEBUG:boto:path=/xryqqk-12/obj41 2024-04-09T21:48:03.947 DEBUG:boto:auth_path=/xryqqk-12/obj41 2024-04-09T21:48:03.947 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.947 DEBUG:boto:Path: /xryqqk-12/obj41 2024-04-09T21:48:03.947 DEBUG:boto:Data: 2024-04-09T21:48:03.947 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.947 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.947 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.947 DEBUG:boto:Params: {} 2024-04-09T21:48:03.947 DEBUG:boto:Token: None 2024-04-09T21:48:03.947 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj41 2024-04-09T21:48:03.948 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:l67q8B5UiIc7mIgCI1vCfj3Ejb0= 2024-04-09T21:48:03.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:l67q8B5UiIc7mIgCI1vCfj3Ejb0='} 2024-04-09T21:48:03.952 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f5bf76cc033c554-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.952 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.952 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.952 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.952 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.953 DEBUG:boto:Data: 2024-04-09T21:48:03.953 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.953 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.953 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.953 DEBUG:boto:Params: {} 2024-04-09T21:48:03.953 DEBUG:boto:Token: None 2024-04-09T21:48:03.953 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/ 2024-04-09T21:48:03.953 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-09T21:48:03.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-09T21:48:03.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000960f758fc9a31117-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.956 DEBUG:boto:path=/xryqqk-12/obj42 2024-04-09T21:48:03.956 DEBUG:boto:auth_path=/xryqqk-12/obj42 2024-04-09T21:48:03.956 DEBUG:boto:Method: PUT 2024-04-09T21:48:03.956 DEBUG:boto:Path: /xryqqk-12/obj42 2024-04-09T21:48:03.956 DEBUG:boto:Data: 2024-04-09T21:48:03.956 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:03.956 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.956 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.956 DEBUG:boto:Params: {} 2024-04-09T21:48:03.956 DEBUG:boto:Token: None 2024-04-09T21:48:03.957 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:03 GMT /xryqqk-12/obj42 2024-04-09T21:48:03.957 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iiIxdEW+p0qlATdDGBTy0ifQzjY= 2024-04-09T21:48:03.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iiIxdEW+p0qlATdDGBTy0ifQzjY='} 2024-04-09T21:48:03.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a49d3c240b4cedf-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:03.999 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:03.999 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:03.999 DEBUG:boto:Method: HEAD 2024-04-09T21:48:03.999 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:03.999 DEBUG:boto:Data: 2024-04-09T21:48:03.999 DEBUG:boto:Headers: {} 2024-04-09T21:48:03.999 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:03.999 DEBUG:boto:Port: 8001 2024-04-09T21:48:03.999 DEBUG:boto:Params: {} 2024-04-09T21:48:04.000 DEBUG:boto:Token: None 2024-04-09T21:48:04.000 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.000 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.002 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b629878ae8601738-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.002 DEBUG:boto:path=/xryqqk-12/obj43 2024-04-09T21:48:04.002 DEBUG:boto:auth_path=/xryqqk-12/obj43 2024-04-09T21:48:04.003 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.003 DEBUG:boto:Path: /xryqqk-12/obj43 2024-04-09T21:48:04.003 DEBUG:boto:Data: 2024-04-09T21:48:04.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.003 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.003 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.003 DEBUG:boto:Params: {} 2024-04-09T21:48:04.003 DEBUG:boto:Token: None 2024-04-09T21:48:04.004 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj43 2024-04-09T21:48:04.004 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kWzkRoFVcU/KpJzaqXD9JxEyE5Y= 2024-04-09T21:48:04.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kWzkRoFVcU/KpJzaqXD9JxEyE5Y='} 2024-04-09T21:48:04.007 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000963c3457bc328f15-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.007 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:04.007 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:04.007 DEBUG:boto:Method: HEAD 2024-04-09T21:48:04.008 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:04.008 DEBUG:boto:Data: 2024-04-09T21:48:04.008 DEBUG:boto:Headers: {} 2024-04-09T21:48:04.008 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.008 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.008 DEBUG:boto:Params: {} 2024-04-09T21:48:04.008 DEBUG:boto:Token: None 2024-04-09T21:48:04.008 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.008 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.010 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7ac1068243e647-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.010 DEBUG:boto:path=/xryqqk-12/obj44 2024-04-09T21:48:04.011 DEBUG:boto:auth_path=/xryqqk-12/obj44 2024-04-09T21:48:04.011 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.011 DEBUG:boto:Path: /xryqqk-12/obj44 2024-04-09T21:48:04.011 DEBUG:boto:Data: 2024-04-09T21:48:04.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.011 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.011 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.011 DEBUG:boto:Params: {} 2024-04-09T21:48:04.011 DEBUG:boto:Token: None 2024-04-09T21:48:04.011 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj44 2024-04-09T21:48:04.012 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M7F1DwLoED2H5cnOQuhzFJ6oOI0= 2024-04-09T21:48:04.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M7F1DwLoED2H5cnOQuhzFJ6oOI0='} 2024-04-09T21:48:04.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000486737cb3f64528a-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.055 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:04.055 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:04.055 DEBUG:boto:Method: HEAD 2024-04-09T21:48:04.055 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:04.055 DEBUG:boto:Data: 2024-04-09T21:48:04.055 DEBUG:boto:Headers: {} 2024-04-09T21:48:04.055 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.055 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.055 DEBUG:boto:Params: {} 2024-04-09T21:48:04.055 DEBUG:boto:Token: None 2024-04-09T21:48:04.056 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.056 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.058 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006008949c268873d7-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.058 DEBUG:boto:path=/xryqqk-12/obj45 2024-04-09T21:48:04.058 DEBUG:boto:auth_path=/xryqqk-12/obj45 2024-04-09T21:48:04.058 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.059 DEBUG:boto:Path: /xryqqk-12/obj45 2024-04-09T21:48:04.059 DEBUG:boto:Data: 2024-04-09T21:48:04.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.059 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.059 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.059 DEBUG:boto:Params: {} 2024-04-09T21:48:04.059 DEBUG:boto:Token: None 2024-04-09T21:48:04.059 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj45 2024-04-09T21:48:04.059 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0QrtXNGUlPNy/r9WZ3ahNfIuyt0= 2024-04-09T21:48:04.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0QrtXNGUlPNy/r9WZ3ahNfIuyt0='} 2024-04-09T21:48:04.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009610bf84f2dad0ea-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.063 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:04.063 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:04.063 DEBUG:boto:Method: HEAD 2024-04-09T21:48:04.063 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:04.063 DEBUG:boto:Data: 2024-04-09T21:48:04.063 DEBUG:boto:Headers: {} 2024-04-09T21:48:04.063 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.064 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.064 DEBUG:boto:Params: {} 2024-04-09T21:48:04.064 DEBUG:boto:Token: None 2024-04-09T21:48:04.064 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.064 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0cc76a81bd5fd48-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.067 DEBUG:boto:path=/xryqqk-12/obj46 2024-04-09T21:48:04.067 DEBUG:boto:auth_path=/xryqqk-12/obj46 2024-04-09T21:48:04.067 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.067 DEBUG:boto:Path: /xryqqk-12/obj46 2024-04-09T21:48:04.067 DEBUG:boto:Data: 2024-04-09T21:48:04.067 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.067 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.067 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.067 DEBUG:boto:Params: {} 2024-04-09T21:48:04.067 DEBUG:boto:Token: None 2024-04-09T21:48:04.067 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj46 2024-04-09T21:48:04.068 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:eJ1bP9mdQpCRfRrchINh+FMpkfs= 2024-04-09T21:48:04.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eJ1bP9mdQpCRfRrchINh+FMpkfs='} 2024-04-09T21:48:04.110 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f961d04f23e547e-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.111 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:04.111 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:04.111 DEBUG:boto:Method: HEAD 2024-04-09T21:48:04.111 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:04.111 DEBUG:boto:Data: 2024-04-09T21:48:04.111 DEBUG:boto:Headers: {} 2024-04-09T21:48:04.111 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.111 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.111 DEBUG:boto:Params: {} 2024-04-09T21:48:04.111 DEBUG:boto:Token: None 2024-04-09T21:48:04.112 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.112 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3c38108a18dff3f-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.114 DEBUG:boto:path=/xryqqk-12/obj47 2024-04-09T21:48:04.114 DEBUG:boto:auth_path=/xryqqk-12/obj47 2024-04-09T21:48:04.114 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.114 DEBUG:boto:Path: /xryqqk-12/obj47 2024-04-09T21:48:04.115 DEBUG:boto:Data: 2024-04-09T21:48:04.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.115 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.115 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.115 DEBUG:boto:Params: {} 2024-04-09T21:48:04.115 DEBUG:boto:Token: None 2024-04-09T21:48:04.115 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj47 2024-04-09T21:48:04.115 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:T8tSR4cESz3YH/nPBMWM8Xi0q6k= 2024-04-09T21:48:04.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T8tSR4cESz3YH/nPBMWM8Xi0q6k='} 2024-04-09T21:48:04.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ede282c02c65d1a4-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.119 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:04.119 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:04.119 DEBUG:boto:Method: HEAD 2024-04-09T21:48:04.119 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:04.119 DEBUG:boto:Data: 2024-04-09T21:48:04.119 DEBUG:boto:Headers: {} 2024-04-09T21:48:04.120 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.120 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.120 DEBUG:boto:Params: {} 2024-04-09T21:48:04.120 DEBUG:boto:Token: None 2024-04-09T21:48:04.120 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.120 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9a28c1e36b39c74-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.123 DEBUG:boto:path=/xryqqk-12/obj48 2024-04-09T21:48:04.123 DEBUG:boto:auth_path=/xryqqk-12/obj48 2024-04-09T21:48:04.123 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.123 DEBUG:boto:Path: /xryqqk-12/obj48 2024-04-09T21:48:04.123 DEBUG:boto:Data: 2024-04-09T21:48:04.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.123 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.123 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.123 DEBUG:boto:Params: {} 2024-04-09T21:48:04.123 DEBUG:boto:Token: None 2024-04-09T21:48:04.124 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj48 2024-04-09T21:48:04.124 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5WA3rl4QkO+gbwWFo4J3zQ1Es5I= 2024-04-09T21:48:04.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5WA3rl4QkO+gbwWFo4J3zQ1Es5I='} 2024-04-09T21:48:04.171 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000211920d2ed0fd997-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.171 DEBUG:boto:path=/xryqqk-12/ 2024-04-09T21:48:04.171 DEBUG:boto:auth_path=/xryqqk-12/ 2024-04-09T21:48:04.171 DEBUG:boto:Method: HEAD 2024-04-09T21:48:04.171 DEBUG:boto:Path: /xryqqk-12/ 2024-04-09T21:48:04.171 DEBUG:boto:Data: 2024-04-09T21:48:04.171 DEBUG:boto:Headers: {} 2024-04-09T21:48:04.172 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.172 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.172 DEBUG:boto:Params: {} 2024-04-09T21:48:04.172 DEBUG:boto:Token: None 2024-04-09T21:48:04.172 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/ 2024-04-09T21:48:04.172 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-09T21:48:04.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-09T21:48:04.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000674f3a5de7c6ab62-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.175 DEBUG:boto:path=/xryqqk-12/obj49 2024-04-09T21:48:04.175 DEBUG:boto:auth_path=/xryqqk-12/obj49 2024-04-09T21:48:04.175 DEBUG:boto:Method: PUT 2024-04-09T21:48:04.175 DEBUG:boto:Path: /xryqqk-12/obj49 2024-04-09T21:48:04.175 DEBUG:boto:Data: 2024-04-09T21:48:04.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T21:48:04.175 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:48:04.175 DEBUG:boto:Port: 8001 2024-04-09T21:48:04.176 DEBUG:boto:Params: {} 2024-04-09T21:48:04.176 DEBUG:boto:Token: None 2024-04-09T21:48:04.176 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 21:48:04 GMT /xryqqk-12/obj49 2024-04-09T21:48:04.176 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:I6noa5m+z9V/d1+M0U6ZDTcCL+U= 2024-04-09T21:48:04.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I6noa5m+z9V/d1+M0U6ZDTcCL+U='} 2024-04-09T21:48:04.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d61c1b24c60733ce-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:48:04.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:48:04.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:48:04.180 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:48:04.228 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:48:04.228 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:48:04.262 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:48:04.274 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:48:04.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.274 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699281.708318_73.1", 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:48:01.708318Z" 2024-04-09T21:48:04.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.276 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:48:04.276 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T21:48:04.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:04.276 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:48:04.277 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:48:04.277 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:48:04.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:48:04.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:48:04.278 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:48:04.327 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:48:04.327 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:48:04.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:04.354+0000 7fe789f86780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:48:04.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:04.355+0000 7fe789f86780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:48:04.368 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:48:04.368 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:48:04.368 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:48:04.368 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:48:04.368 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:48:04.368 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:48:04.369 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:04.370 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:48:04.370 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:04.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:04.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.370 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:04.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.343778_63.1", 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.343778Z", 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:04.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:04.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:48:04Z" 2024-04-09T21:48:04.374 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:48:04.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699220.343778_63.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.343778Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:48:04Z" } 2024-04-09T21:48:04.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699220.343778_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.343778Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:48:04.375 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:48:04.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699220.343778_63.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:48:04.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712699281.708318_73.1 target=1_1712699220.343778_63.1 2024-04-09T21:48:09.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:48:09.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:48:09.377 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:48:09.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:48:09.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:48:09.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:09.467+0000 7fe6a0534780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:48:09.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:09.468+0000 7fe6a0534780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:48:09.482 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:09.483 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:09.484 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:09.485 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699281.708318_73.1", 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:48:01.708318Z", 2024-04-09T21:48:09.486 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:48:09.487 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:48:09Z" 2024-04-09T21:48:09.488 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:48:09.489 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699281.708318_73.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:48:01.708318Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:48:09Z" } 2024-04-09T21:48:09.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699281.708318_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:48:01.708318Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:48:09.490 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:48:09.490 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:48:09.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:48:09.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:48:09.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:48:09.490 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:48:09.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:48:09.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:48:09.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:09.583+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:14.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:14.584+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:19.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:19.585+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:24.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:24.584+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:29.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:29.585+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:34.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:34.586+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:39.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:39.586+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:44.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:44.587+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:49.587+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:54.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:54.588+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:48:59.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:48:59.589+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:04.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:04.589+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:09.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:09.590+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:14.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:14.591+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:19.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:19.591+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:24.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:24.592+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:29.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:29.593+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:34.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:34.593+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:39.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:39.594+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:44.595+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:49.595+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:54.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:54.597+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:49:59.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:49:59.598+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:04.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:04.597+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:09.598+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:14.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:14.599+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:19.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:19.599+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:24.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:24.600+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:29.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:29.601+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:34.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:34.601+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:39.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:39.602+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:44.602+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:49.603+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:54.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:54.604+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:50:59.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:50:59.604+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:04.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:04.605+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:09.606+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:14.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:14.607+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:19.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:19.607+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:24.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:24.608+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:29.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:29.609+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:34.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:34.610+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:39.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:39.611+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:44.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:44.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:49.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:49.611+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:54.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:54.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:51:59.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:51:59.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:04.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:04.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:09.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:09.613+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:14.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:14.614+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:19.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:19.614+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:24.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:24.615+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:29.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:29.616+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:34.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:34.616+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-09T21:52:39.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:39.618+0000 7fe38fc07780 1 bucket sync caught up with source: 2024-04-09T21:52:39.621 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [00000000007.24.6, 00000000004.12.6, 00000000004.14.6, 00000000004.13.6, 00000000006.13.6, 00000000005.20.6, 00000000004.19.6, 00000000005.14.6, 00000000002.7.6, 00000000004.11.6, 00000000004.17.6] 2024-04-09T21:52:39.621 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [00000000007.24.6, 00000000004.12.6, 00000000004.14.6, 00000000004.13.6, 00000000006.13.6, 00000000005.20.6, 00000000004.19.6, 00000000005.14.6, 00000000002.7.6, 00000000004.11.6, 00000000004.17.6] 2024-04-09T21:52:39.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:39.618+0000 7fe38fc07780 0 bucket checkpoint complete 2024-04-09T21:52:39.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-11 zones={a2, a1} 2024-04-09T21:52:39.638 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.638 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.639 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.639 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:52:39.639 DEBUG:boto:Data: 2024-04-09T21:52:39.639 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.639 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.639 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.639 DEBUG:boto:Params: {} 2024-04-09T21:52:39.640 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T21:52:39.640 DEBUG:boto:Token: None 2024-04-09T21:52:39.640 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.640 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM= 2024-04-09T21:52:39.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM=', 'Content-Length': '0'} 2024-04-09T21:52:39.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036fcdb8d49719b1a-006615b8a7-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.645 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.646 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.646 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.646 DEBUG:boto:Path: /xryqqk-11/ 2024-04-09T21:52:39.646 DEBUG:boto:Data: 2024-04-09T21:52:39.646 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.646 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.646 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.646 DEBUG:boto:Params: {} 2024-04-09T21:52:39.646 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T21:52:39.647 DEBUG:boto:Token: None 2024-04-09T21:52:39.647 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.647 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM= 2024-04-09T21:52:39.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM=', 'Content-Length': '0'} 2024-04-09T21:52:39.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070c0d2ba4394154f-006615b8a7-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:52:39.652 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.652 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.652 DEBUG:boto:path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.652 DEBUG:boto:auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.652 DEBUG:boto:Method: GET 2024-04-09T21:52:39.652 DEBUG:boto:Path: /xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.652 DEBUG:boto:Data: 2024-04-09T21:52:39.652 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.652 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.652 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.653 DEBUG:boto:Params: {} 2024-04-09T21:52:39.653 DEBUG:boto:Token: None 2024-04-09T21:52:39.653 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/?versions 2024-04-09T21:52:39.653 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-09T21:52:39.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-09T21:52:39.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b071671708afbde-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.656 DEBUG:boto:b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-09T21:52:39.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj10 2024-04-09T21:52:39.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-09T21:52:39.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj12 2024-04-09T21:52:39.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj13 2024-04-09T21:52:39.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj14 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj15 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj16 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj18 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj20 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj21 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj24 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj25 2024-04-09T21:52:39.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj26 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj27 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj28 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj30 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj32 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj35 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj36 2024-04-09T21:52:39.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj39 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj40 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj41 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj42 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj43 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj45 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj47 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj48 2024-04-09T21:52:39.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj49 2024-04-09T21:52:39.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-09T21:52:39.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-09T21:52:39.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-09T21:52:39.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-09T21:52:39.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj9 2024-04-09T21:52:39.663 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:52:39.663 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.663 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.663 DEBUG:boto:path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.663 DEBUG:boto:auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.664 DEBUG:boto:Method: GET 2024-04-09T21:52:39.664 DEBUG:boto:Path: /xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.664 DEBUG:boto:Data: 2024-04-09T21:52:39.664 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.664 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.664 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.664 DEBUG:boto:Params: {} 2024-04-09T21:52:39.664 DEBUG:boto:Token: None 2024-04-09T21:52:39.664 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/?versions 2024-04-09T21:52:39.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-09T21:52:39.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-09T21:52:39.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab14618761cf4457-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.667 DEBUG:boto:b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj10 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj12 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj13 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj14 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj15 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj16 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-09T21:52:39.670 DEBUG:tasks.rgw_multi.zone_rados:o=obj18 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj20 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj21 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj24 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj25 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj26 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj27 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj28 2024-04-09T21:52:39.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj30 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj32 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj35 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj36 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj39 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-09T21:52:39.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj40 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj41 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj42 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj43 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj45 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj47 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj48 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj49 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-09T21:52:39.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-09T21:52:39.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-09T21:52:39.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj9 2024-04-09T21:52:39.674 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.674 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.674 DEBUG:boto:path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.674 DEBUG:boto:auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.674 DEBUG:boto:Method: GET 2024-04-09T21:52:39.674 DEBUG:boto:Path: /xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.674 DEBUG:boto:Data: 2024-04-09T21:52:39.674 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.675 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.675 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.675 DEBUG:boto:Params: {} 2024-04-09T21:52:39.675 DEBUG:boto:Token: None 2024-04-09T21:52:39.675 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/?versions 2024-04-09T21:52:39.675 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-09T21:52:39.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-09T21:52:39.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a08dd81300de12-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.678 DEBUG:boto:b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.680 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.681 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.681 DEBUG:boto:path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.681 DEBUG:boto:auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.681 DEBUG:boto:Method: GET 2024-04-09T21:52:39.681 DEBUG:boto:Path: /xryqqk-11/?versions&max-keys=999 2024-04-09T21:52:39.681 DEBUG:boto:Data: 2024-04-09T21:52:39.681 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.681 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.681 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.681 DEBUG:boto:Params: {} 2024-04-09T21:52:39.681 DEBUG:boto:Token: None 2024-04-09T21:52:39.682 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/?versions 2024-04-09T21:52:39.682 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-09T21:52:39.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-09T21:52:39.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ba8b2917225c93b-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.689 DEBUG:boto:b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-09T21:52:39.692 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.692 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.692 DEBUG:boto:path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.692 DEBUG:boto:auth_path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.692 DEBUG:boto:Method: GET 2024-04-09T21:52:39.692 DEBUG:boto:Path: /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.692 DEBUG:boto:Data: 2024-04-09T21:52:39.693 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.693 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.693 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.693 DEBUG:boto:Params: {} 2024-04-09T21:52:39.693 DEBUG:boto:Token: None 2024-04-09T21:52:39.693 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.693 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-09T21:52:39.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-09T21:52:39.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000261bfdac56e7021a-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.696 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.696 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.696 DEBUG:boto:path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.696 DEBUG:boto:auth_path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.696 DEBUG:boto:Method: GET 2024-04-09T21:52:39.696 DEBUG:boto:Path: /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.696 DEBUG:boto:Data: 2024-04-09T21:52:39.696 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.696 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.696 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.696 DEBUG:boto:Params: {} 2024-04-09T21:52:39.697 DEBUG:boto:Token: None 2024-04-09T21:52:39.697 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.697 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-09T21:52:39.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-09T21:52:39.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f01c93bc07dfb0c5-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.746 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.747 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.747 DEBUG:boto:path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.747 DEBUG:boto:auth_path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.747 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.747 DEBUG:boto:Path: /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.747 DEBUG:boto:Data: 2024-04-09T21:52:39.747 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.747 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.747 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.747 DEBUG:boto:Params: {} 2024-04-09T21:52:39.748 DEBUG:boto:Token: None 2024-04-09T21:52:39.748 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.748 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg= 2024-04-09T21:52:39.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg=', 'Content-Length': '0'} 2024-04-09T21:52:39.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fff52879f097f9f2-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.750 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.750 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.750 DEBUG:boto:path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.750 DEBUG:boto:auth_path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.750 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.750 DEBUG:boto:Path: /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.751 DEBUG:boto:Data: 2024-04-09T21:52:39.751 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.751 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.751 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.751 DEBUG:boto:Params: {} 2024-04-09T21:52:39.751 DEBUG:boto:Token: None 2024-04-09T21:52:39.751 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.751 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg= 2024-04-09T21:52:39.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg=', 'Content-Length': '0'} 2024-04-09T21:52:39.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e8064bb04a92ffd-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-09T21:52:39.753 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.753 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.753 DEBUG:boto:path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.754 DEBUG:boto:auth_path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.754 DEBUG:boto:Method: GET 2024-04-09T21:52:39.754 DEBUG:boto:Path: /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.754 DEBUG:boto:Data: 2024-04-09T21:52:39.754 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.754 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.754 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.754 DEBUG:boto:Params: {} 2024-04-09T21:52:39.754 DEBUG:boto:Token: None 2024-04-09T21:52:39.755 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.755 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-09T21:52:39.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-09T21:52:39.756 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e2d93598b950d37-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.757 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.757 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.757 DEBUG:boto:path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.757 DEBUG:boto:auth_path=/xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.757 DEBUG:boto:Method: GET 2024-04-09T21:52:39.757 DEBUG:boto:Path: /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.757 DEBUG:boto:Data: 2024-04-09T21:52:39.758 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.758 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.758 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.758 DEBUG:boto:Params: {} 2024-04-09T21:52:39.758 DEBUG:boto:Token: None 2024-04-09T21:52:39.758 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1?versionId=null 2024-04-09T21:52:39.758 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-09T21:52:39.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-09T21:52:39.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aad22a58ec773d2-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.760 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.760 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.761 DEBUG:boto:path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.761 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.761 DEBUG:boto:Method: GET 2024-04-09T21:52:39.761 DEBUG:boto:Path: /xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.761 DEBUG:boto:Data: 2024-04-09T21:52:39.761 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.761 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.761 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.761 DEBUG:boto:Params: {} 2024-04-09T21:52:39.761 DEBUG:boto:Token: None 2024-04-09T21:52:39.762 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.762 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd683c5937065fe8-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.764 DEBUG:boto:b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.765 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.765 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.765 DEBUG:boto:path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.765 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.765 DEBUG:boto:Method: GET 2024-04-09T21:52:39.766 DEBUG:boto:Path: /xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.766 DEBUG:boto:Data: 2024-04-09T21:52:39.766 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.766 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.766 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.766 DEBUG:boto:Params: {} 2024-04-09T21:52:39.766 DEBUG:boto:Token: None 2024-04-09T21:52:39.766 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.766 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000900d30586391af9-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.769 DEBUG:boto:b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.770 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.770 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.770 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.770 DEBUG:boto:Path: /xryqqk-11/obj1 2024-04-09T21:52:39.770 DEBUG:boto:Data: 2024-04-09T21:52:39.770 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.770 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.770 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.770 DEBUG:boto:Params: {} 2024-04-09T21:52:39.771 DEBUG:boto:Token: None 2024-04-09T21:52:39.771 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1 2024-04-09T21:52:39.771 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko= 2024-04-09T21:52:39.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko=', 'Content-Length': '0'} 2024-04-09T21:52:39.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f375078042aeb881-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.773 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.773 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.773 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.773 DEBUG:boto:Path: /xryqqk-11/obj1 2024-04-09T21:52:39.773 DEBUG:boto:Data: 2024-04-09T21:52:39.773 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.773 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.773 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.773 DEBUG:boto:Params: {} 2024-04-09T21:52:39.774 DEBUG:boto:Token: None 2024-04-09T21:52:39.774 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1 2024-04-09T21:52:39.774 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko= 2024-04-09T21:52:39.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko=', 'Content-Length': '0'} 2024-04-09T21:52:39.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3a0f62e0a841cdc-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-09T21:52:39.776 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.776 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.776 DEBUG:boto:Method: GET 2024-04-09T21:52:39.776 DEBUG:boto:Path: /xryqqk-11/obj1 2024-04-09T21:52:39.776 DEBUG:boto:Data: 2024-04-09T21:52:39.776 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.776 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.776 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.776 DEBUG:boto:Params: {} 2024-04-09T21:52:39.777 DEBUG:boto:Token: None 2024-04-09T21:52:39.777 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1 2024-04-09T21:52:39.777 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE= 2024-04-09T21:52:39.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE=', 'Content-Length': '0'} 2024-04-09T21:52:39.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e35cbbb42d04486-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.779 DEBUG:boto:path=/xryqqk-11/obj1 2024-04-09T21:52:39.779 DEBUG:boto:auth_path=/xryqqk-11/obj1 2024-04-09T21:52:39.779 DEBUG:boto:Method: GET 2024-04-09T21:52:39.779 DEBUG:boto:Path: /xryqqk-11/obj1 2024-04-09T21:52:39.779 DEBUG:boto:Data: 2024-04-09T21:52:39.779 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.779 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.779 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.779 DEBUG:boto:Params: {} 2024-04-09T21:52:39.780 DEBUG:boto:Token: None 2024-04-09T21:52:39.780 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj1 2024-04-09T21:52:39.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE= 2024-04-09T21:52:39.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE=', 'Content-Length': '0'} 2024-04-09T21:52:39.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e38579db120a553-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.782 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.782 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.782 DEBUG:boto:path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.782 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.782 DEBUG:boto:Method: GET 2024-04-09T21:52:39.782 DEBUG:boto:Path: /xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.782 DEBUG:boto:Data: 2024-04-09T21:52:39.783 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.783 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.783 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.783 DEBUG:boto:Params: {} 2024-04-09T21:52:39.783 DEBUG:boto:Token: None 2024-04-09T21:52:39.783 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.783 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecac3bb40c5a22ee-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.786 DEBUG:boto:b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.787 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.787 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.787 DEBUG:boto:path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.787 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.787 DEBUG:boto:Method: GET 2024-04-09T21:52:39.787 DEBUG:boto:Path: /xryqqk-11/?prefix=obj1 2024-04-09T21:52:39.787 DEBUG:boto:Data: 2024-04-09T21:52:39.787 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.787 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.787 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.788 DEBUG:boto:Params: {} 2024-04-09T21:52:39.788 DEBUG:boto:Token: None 2024-04-09T21:52:39.788 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.788 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a554e00439430e6-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.790 DEBUG:boto:b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-04-09T21:52:39.791 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.791 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.791 DEBUG:boto:path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.791 DEBUG:boto:auth_path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.792 DEBUG:boto:Method: GET 2024-04-09T21:52:39.792 DEBUG:boto:Path: /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.792 DEBUG:boto:Data: 2024-04-09T21:52:39.792 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.792 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.792 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.792 DEBUG:boto:Params: {} 2024-04-09T21:52:39.792 DEBUG:boto:Token: None 2024-04-09T21:52:39.792 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.792 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-09T21:52:39.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-09T21:52:39.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb959997e6da8f40-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.842 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.842 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.842 DEBUG:boto:path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.843 DEBUG:boto:auth_path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.843 DEBUG:boto:Method: GET 2024-04-09T21:52:39.843 DEBUG:boto:Path: /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.843 DEBUG:boto:Data: 2024-04-09T21:52:39.843 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.843 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.843 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.843 DEBUG:boto:Params: {} 2024-04-09T21:52:39.843 DEBUG:boto:Token: None 2024-04-09T21:52:39.844 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.844 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-09T21:52:39.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-09T21:52:39.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e8946ef32fc87b5-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.846 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.846 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.846 DEBUG:boto:path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.846 DEBUG:boto:auth_path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.846 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.846 DEBUG:boto:Path: /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.846 DEBUG:boto:Data: 2024-04-09T21:52:39.846 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.846 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.847 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.847 DEBUG:boto:Params: {} 2024-04-09T21:52:39.847 DEBUG:boto:Token: None 2024-04-09T21:52:39.847 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.847 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM= 2024-04-09T21:52:39.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM=', 'Content-Length': '0'} 2024-04-09T21:52:39.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c51b7c800aaad7ab-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.849 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.849 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.849 DEBUG:boto:path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.849 DEBUG:boto:auth_path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.849 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.849 DEBUG:boto:Path: /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.849 DEBUG:boto:Data: 2024-04-09T21:52:39.850 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.850 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.850 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.850 DEBUG:boto:Params: {} 2024-04-09T21:52:39.850 DEBUG:boto:Token: None 2024-04-09T21:52:39.850 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.850 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM= 2024-04-09T21:52:39.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM=', 'Content-Length': '0'} 2024-04-09T21:52:39.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaa45d75ef5a28ed-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-04-09T21:52:39.852 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.852 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.852 DEBUG:boto:path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.853 DEBUG:boto:auth_path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.853 DEBUG:boto:Method: GET 2024-04-09T21:52:39.853 DEBUG:boto:Path: /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.853 DEBUG:boto:Data: 2024-04-09T21:52:39.853 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.853 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.853 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.853 DEBUG:boto:Params: {} 2024-04-09T21:52:39.853 DEBUG:boto:Token: None 2024-04-09T21:52:39.853 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.854 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-09T21:52:39.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-09T21:52:39.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000427003c4064fad66-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.855 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.855 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.856 DEBUG:boto:path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.856 DEBUG:boto:auth_path=/xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.856 DEBUG:boto:Method: GET 2024-04-09T21:52:39.856 DEBUG:boto:Path: /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.856 DEBUG:boto:Data: 2024-04-09T21:52:39.856 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.856 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.856 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.856 DEBUG:boto:Params: {} 2024-04-09T21:52:39.856 DEBUG:boto:Token: None 2024-04-09T21:52:39.857 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10?versionId=null 2024-04-09T21:52:39.857 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-09T21:52:39.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-09T21:52:39.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a48291995e421728-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.858 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.859 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.859 DEBUG:boto:path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.859 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.859 DEBUG:boto:Method: GET 2024-04-09T21:52:39.859 DEBUG:boto:Path: /xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.859 DEBUG:boto:Data: 2024-04-09T21:52:39.859 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.859 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.859 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.859 DEBUG:boto:Params: {} 2024-04-09T21:52:39.859 DEBUG:boto:Token: None 2024-04-09T21:52:39.860 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.860 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081622a2aa1f746a-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.863 DEBUG:boto:b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.863 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.863 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.863 DEBUG:boto:path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.863 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.864 DEBUG:boto:Method: GET 2024-04-09T21:52:39.864 DEBUG:boto:Path: /xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.864 DEBUG:boto:Data: 2024-04-09T21:52:39.864 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.864 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.864 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.864 DEBUG:boto:Params: {} 2024-04-09T21:52:39.864 DEBUG:boto:Token: None 2024-04-09T21:52:39.864 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.864 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004972b9a9702cb96e-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.867 DEBUG:boto:b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.867 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.867 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.867 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.868 DEBUG:boto:Path: /xryqqk-11/obj10 2024-04-09T21:52:39.868 DEBUG:boto:Data: 2024-04-09T21:52:39.868 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.868 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.868 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.868 DEBUG:boto:Params: {} 2024-04-09T21:52:39.868 DEBUG:boto:Token: None 2024-04-09T21:52:39.868 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10 2024-04-09T21:52:39.868 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM= 2024-04-09T21:52:39.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM=', 'Content-Length': '0'} 2024-04-09T21:52:39.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f939583462ba2e73-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.870 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.870 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.870 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.870 DEBUG:boto:Path: /xryqqk-11/obj10 2024-04-09T21:52:39.870 DEBUG:boto:Data: 2024-04-09T21:52:39.870 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.871 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.871 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.871 DEBUG:boto:Params: {} 2024-04-09T21:52:39.871 DEBUG:boto:Token: None 2024-04-09T21:52:39.871 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10 2024-04-09T21:52:39.871 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM= 2024-04-09T21:52:39.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM=', 'Content-Length': '0'} 2024-04-09T21:52:39.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c69767a0e317f8-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj10 2024-04-09T21:52:39.873 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.873 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.873 DEBUG:boto:Method: GET 2024-04-09T21:52:39.873 DEBUG:boto:Path: /xryqqk-11/obj10 2024-04-09T21:52:39.873 DEBUG:boto:Data: 2024-04-09T21:52:39.873 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.873 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.874 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.874 DEBUG:boto:Params: {} 2024-04-09T21:52:39.874 DEBUG:boto:Token: None 2024-04-09T21:52:39.874 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10 2024-04-09T21:52:39.874 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ= 2024-04-09T21:52:39.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ=', 'Content-Length': '0'} 2024-04-09T21:52:39.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073ff9e68d5f54494-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.919 DEBUG:boto:path=/xryqqk-11/obj10 2024-04-09T21:52:39.919 DEBUG:boto:auth_path=/xryqqk-11/obj10 2024-04-09T21:52:39.919 DEBUG:boto:Method: GET 2024-04-09T21:52:39.919 DEBUG:boto:Path: /xryqqk-11/obj10 2024-04-09T21:52:39.919 DEBUG:boto:Data: 2024-04-09T21:52:39.919 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.919 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.920 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.920 DEBUG:boto:Params: {} 2024-04-09T21:52:39.920 DEBUG:boto:Token: None 2024-04-09T21:52:39.920 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj10 2024-04-09T21:52:39.920 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ= 2024-04-09T21:52:39.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ=', 'Content-Length': '0'} 2024-04-09T21:52:39.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007243b43d2ccf0e14-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.922 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.922 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.922 DEBUG:boto:path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.923 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.923 DEBUG:boto:Method: GET 2024-04-09T21:52:39.923 DEBUG:boto:Path: /xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.923 DEBUG:boto:Data: 2024-04-09T21:52:39.923 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.923 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.923 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.923 DEBUG:boto:Params: {} 2024-04-09T21:52:39.924 DEBUG:boto:Token: None 2024-04-09T21:52:39.924 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.924 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588e653cf6f8dc10-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.926 DEBUG:boto:b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.927 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.927 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.927 DEBUG:boto:path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.927 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.927 DEBUG:boto:Method: GET 2024-04-09T21:52:39.927 DEBUG:boto:Path: /xryqqk-11/?prefix=obj10 2024-04-09T21:52:39.927 DEBUG:boto:Data: 2024-04-09T21:52:39.928 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.928 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.928 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.928 DEBUG:boto:Params: {} 2024-04-09T21:52:39.928 DEBUG:boto:Token: None 2024-04-09T21:52:39.928 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:39.928 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:39.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:39.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b61feb16fa9c7ba-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.930 DEBUG:boto:b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:39.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-09T21:52:39.931 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:39.931 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:39.931 DEBUG:boto:path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.931 DEBUG:boto:auth_path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.931 DEBUG:boto:Method: GET 2024-04-09T21:52:39.931 DEBUG:boto:Path: /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.931 DEBUG:boto:Data: 2024-04-09T21:52:39.932 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.932 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.932 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.932 DEBUG:boto:Params: {} 2024-04-09T21:52:39.932 DEBUG:boto:Token: None 2024-04-09T21:52:39.932 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.932 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-09T21:52:39.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-09T21:52:39.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0a4802e3d286044-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.934 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:39.935 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:39.935 DEBUG:boto:path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.935 DEBUG:boto:auth_path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.935 DEBUG:boto:Method: GET 2024-04-09T21:52:39.935 DEBUG:boto:Path: /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.935 DEBUG:boto:Data: 2024-04-09T21:52:39.935 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.935 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.935 DEBUG:boto:Params: {} 2024-04-09T21:52:39.936 DEBUG:boto:Token: None 2024-04-09T21:52:39.936 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-09T21:52:39.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-09T21:52:39.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000589dfde47698b30b-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.938 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:39.938 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:39.938 DEBUG:boto:path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.938 DEBUG:boto:auth_path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.938 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.938 DEBUG:boto:Path: /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.938 DEBUG:boto:Data: 2024-04-09T21:52:39.938 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.939 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.939 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.939 DEBUG:boto:Params: {} 2024-04-09T21:52:39.939 DEBUG:boto:Token: None 2024-04-09T21:52:39.939 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA= 2024-04-09T21:52:39.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA=', 'Content-Length': '0'} 2024-04-09T21:52:39.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcc8f7fa68ffacbe-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.941 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:39.941 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:39.941 DEBUG:boto:path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.941 DEBUG:boto:auth_path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.941 DEBUG:boto:Method: HEAD 2024-04-09T21:52:39.941 DEBUG:boto:Path: /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.941 DEBUG:boto:Data: 2024-04-09T21:52:39.941 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.942 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.942 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.942 DEBUG:boto:Params: {} 2024-04-09T21:52:39.942 DEBUG:boto:Token: None 2024-04-09T21:52:39.942 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.942 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA= 2024-04-09T21:52:39.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA=', 'Content-Length': '0'} 2024-04-09T21:52:39.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1e44a993e2de720-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-09T21:52:39.944 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:39.944 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:39.944 DEBUG:boto:path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.944 DEBUG:boto:auth_path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.944 DEBUG:boto:Method: GET 2024-04-09T21:52:39.945 DEBUG:boto:Path: /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.945 DEBUG:boto:Data: 2024-04-09T21:52:39.945 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.945 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.945 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.945 DEBUG:boto:Params: {} 2024-04-09T21:52:39.945 DEBUG:boto:Token: None 2024-04-09T21:52:39.945 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.945 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-09T21:52:39.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-09T21:52:39.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc0059f28623e5a0-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.994 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:39.995 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:39.995 DEBUG:boto:path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.995 DEBUG:boto:auth_path=/xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.995 DEBUG:boto:Method: GET 2024-04-09T21:52:39.995 DEBUG:boto:Path: /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.995 DEBUG:boto:Data: 2024-04-09T21:52:39.995 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.995 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:39.995 DEBUG:boto:Port: 8000 2024-04-09T21:52:39.995 DEBUG:boto:Params: {} 2024-04-09T21:52:39.995 DEBUG:boto:Token: None 2024-04-09T21:52:39.996 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/obj11?versionId=null 2024-04-09T21:52:39.996 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-09T21:52:39.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-09T21:52:39.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049075f72e1fac1bf-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:39.998 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:39.998 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:39.998 DEBUG:boto:path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:39.999 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:39.999 DEBUG:boto:Method: GET 2024-04-09T21:52:39.999 DEBUG:boto:Path: /xryqqk-11/?prefix=obj11 2024-04-09T21:52:39.999 DEBUG:boto:Data: 2024-04-09T21:52:39.999 DEBUG:boto:Headers: {} 2024-04-09T21:52:39.999 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:39.999 DEBUG:boto:Port: 8001 2024-04-09T21:52:39.999 DEBUG:boto:Params: {} 2024-04-09T21:52:39.999 DEBUG:boto:Token: None 2024-04-09T21:52:39.999 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:39 GMT /xryqqk-11/ 2024-04-09T21:52:40.000 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-09T21:52:40.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-09T21:52:40.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48b041ff34eeb57-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.002 DEBUG:boto:b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.003 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.003 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.003 DEBUG:boto:path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.003 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.003 DEBUG:boto:Method: GET 2024-04-09T21:52:40.003 DEBUG:boto:Path: /xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.003 DEBUG:boto:Data: 2024-04-09T21:52:40.003 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.003 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.003 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.004 DEBUG:boto:Params: {} 2024-04-09T21:52:40.004 DEBUG:boto:Token: None 2024-04-09T21:52:40.004 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.004 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfefbb2468133c3d-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.007 DEBUG:boto:b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.008 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:40.008 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:40.008 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.008 DEBUG:boto:Path: /xryqqk-11/obj11 2024-04-09T21:52:40.008 DEBUG:boto:Data: 2024-04-09T21:52:40.009 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.009 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.009 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.009 DEBUG:boto:Params: {} 2024-04-09T21:52:40.010 DEBUG:boto:Token: None 2024-04-09T21:52:40.010 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj11 2024-04-09T21:52:40.010 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw= 2024-04-09T21:52:40.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw=', 'Content-Length': '0'} 2024-04-09T21:52:40.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002339967d26a81e8f-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.014 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:40.014 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:40.014 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.014 DEBUG:boto:Path: /xryqqk-11/obj11 2024-04-09T21:52:40.015 DEBUG:boto:Data: 2024-04-09T21:52:40.015 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.015 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.015 DEBUG:boto:Params: {} 2024-04-09T21:52:40.016 DEBUG:boto:Token: None 2024-04-09T21:52:40.016 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj11 2024-04-09T21:52:40.016 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw= 2024-04-09T21:52:40.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw=', 'Content-Length': '0'} 2024-04-09T21:52:40.018 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014632a78a36646e4-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-09T21:52:40.019 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:40.019 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:40.019 DEBUG:boto:Method: GET 2024-04-09T21:52:40.020 DEBUG:boto:Path: /xryqqk-11/obj11 2024-04-09T21:52:40.020 DEBUG:boto:Data: 2024-04-09T21:52:40.020 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.020 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.020 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.021 DEBUG:boto:Params: {} 2024-04-09T21:52:40.021 DEBUG:boto:Token: None 2024-04-09T21:52:40.021 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj11 2024-04-09T21:52:40.021 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o= 2024-04-09T21:52:40.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o=', 'Content-Length': '0'} 2024-04-09T21:52:40.024 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdc09791e73f5c9f-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.024 DEBUG:boto:path=/xryqqk-11/obj11 2024-04-09T21:52:40.025 DEBUG:boto:auth_path=/xryqqk-11/obj11 2024-04-09T21:52:40.025 DEBUG:boto:Method: GET 2024-04-09T21:52:40.025 DEBUG:boto:Path: /xryqqk-11/obj11 2024-04-09T21:52:40.025 DEBUG:boto:Data: 2024-04-09T21:52:40.025 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.025 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.025 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.025 DEBUG:boto:Params: {} 2024-04-09T21:52:40.026 DEBUG:boto:Token: None 2024-04-09T21:52:40.026 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj11 2024-04-09T21:52:40.026 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o= 2024-04-09T21:52:40.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o=', 'Content-Length': '0'} 2024-04-09T21:52:40.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d53a1f9861040d2-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.029 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.029 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.029 DEBUG:boto:path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.029 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.029 DEBUG:boto:Method: GET 2024-04-09T21:52:40.029 DEBUG:boto:Path: /xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.029 DEBUG:boto:Data: 2024-04-09T21:52:40.030 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.030 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.030 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.030 DEBUG:boto:Params: {} 2024-04-09T21:52:40.030 DEBUG:boto:Token: None 2024-04-09T21:52:40.030 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.030 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d56a248d017aa70-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.033 DEBUG:boto:b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.033 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.033 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.033 DEBUG:boto:path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.034 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.034 DEBUG:boto:Method: GET 2024-04-09T21:52:40.034 DEBUG:boto:Path: /xryqqk-11/?prefix=obj11 2024-04-09T21:52:40.034 DEBUG:boto:Data: 2024-04-09T21:52:40.034 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.034 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.034 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.034 DEBUG:boto:Params: {} 2024-04-09T21:52:40.034 DEBUG:boto:Token: None 2024-04-09T21:52:40.034 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.035 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd50c23e0cc1f78a-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.037 DEBUG:boto:b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-04-09T21:52:40.037 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.037 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.038 DEBUG:boto:path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.038 DEBUG:boto:auth_path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.038 DEBUG:boto:Method: GET 2024-04-09T21:52:40.038 DEBUG:boto:Path: /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.038 DEBUG:boto:Data: 2024-04-09T21:52:40.038 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.038 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.038 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.038 DEBUG:boto:Params: {} 2024-04-09T21:52:40.038 DEBUG:boto:Token: None 2024-04-09T21:52:40.039 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.039 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-09T21:52:40.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-09T21:52:40.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008154156de9478a59-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.041 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.041 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.041 DEBUG:boto:path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.041 DEBUG:boto:auth_path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.041 DEBUG:boto:Method: GET 2024-04-09T21:52:40.041 DEBUG:boto:Path: /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.041 DEBUG:boto:Data: 2024-04-09T21:52:40.041 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.041 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.042 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.042 DEBUG:boto:Params: {} 2024-04-09T21:52:40.042 DEBUG:boto:Token: None 2024-04-09T21:52:40.042 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.042 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-09T21:52:40.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-09T21:52:40.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d18179685090f918-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.044 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.044 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.044 DEBUG:boto:path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.044 DEBUG:boto:auth_path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.045 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.045 DEBUG:boto:Path: /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.045 DEBUG:boto:Data: 2024-04-09T21:52:40.045 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.045 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.045 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.045 DEBUG:boto:Params: {} 2024-04-09T21:52:40.045 DEBUG:boto:Token: None 2024-04-09T21:52:40.045 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.045 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk= 2024-04-09T21:52:40.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk=', 'Content-Length': '0'} 2024-04-09T21:52:40.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073b255d4b37a8d7d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.047 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.047 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.047 DEBUG:boto:path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.047 DEBUG:boto:auth_path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.048 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.048 DEBUG:boto:Path: /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.048 DEBUG:boto:Data: 2024-04-09T21:52:40.048 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.048 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.048 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.048 DEBUG:boto:Params: {} 2024-04-09T21:52:40.048 DEBUG:boto:Token: None 2024-04-09T21:52:40.048 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.048 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk= 2024-04-09T21:52:40.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk=', 'Content-Length': '0'} 2024-04-09T21:52:40.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b717cd859b0f8215-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-04-09T21:52:40.050 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.050 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.051 DEBUG:boto:path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.051 DEBUG:boto:auth_path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.051 DEBUG:boto:Method: GET 2024-04-09T21:52:40.051 DEBUG:boto:Path: /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.051 DEBUG:boto:Data: 2024-04-09T21:52:40.051 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.051 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.051 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.051 DEBUG:boto:Params: {} 2024-04-09T21:52:40.051 DEBUG:boto:Token: None 2024-04-09T21:52:40.051 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.052 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-09T21:52:40.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-09T21:52:40.053 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ef030979a4c97cb-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.094 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.095 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.095 DEBUG:boto:path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.095 DEBUG:boto:auth_path=/xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.095 DEBUG:boto:Method: GET 2024-04-09T21:52:40.095 DEBUG:boto:Path: /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.095 DEBUG:boto:Data: 2024-04-09T21:52:40.095 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.095 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.095 DEBUG:boto:Params: {} 2024-04-09T21:52:40.096 DEBUG:boto:Token: None 2024-04-09T21:52:40.096 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12?versionId=null 2024-04-09T21:52:40.096 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-09T21:52:40.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-09T21:52:40.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce3f22a86cffa240-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.098 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.098 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.098 DEBUG:boto:path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.098 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.098 DEBUG:boto:Method: GET 2024-04-09T21:52:40.098 DEBUG:boto:Path: /xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.099 DEBUG:boto:Data: 2024-04-09T21:52:40.099 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.099 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.099 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.099 DEBUG:boto:Params: {} 2024-04-09T21:52:40.099 DEBUG:boto:Token: None 2024-04-09T21:52:40.099 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.099 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005313c35820addb3f-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.102 DEBUG:boto:b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.102 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.102 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.102 DEBUG:boto:path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.102 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.103 DEBUG:boto:Method: GET 2024-04-09T21:52:40.103 DEBUG:boto:Path: /xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.103 DEBUG:boto:Data: 2024-04-09T21:52:40.103 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.103 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.103 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.103 DEBUG:boto:Params: {} 2024-04-09T21:52:40.103 DEBUG:boto:Token: None 2024-04-09T21:52:40.103 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.103 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a5e46db1d9be9b-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.106 DEBUG:boto:b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.106 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.106 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.106 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.107 DEBUG:boto:Path: /xryqqk-11/obj12 2024-04-09T21:52:40.107 DEBUG:boto:Data: 2024-04-09T21:52:40.107 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.107 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.107 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.107 DEBUG:boto:Params: {} 2024-04-09T21:52:40.107 DEBUG:boto:Token: None 2024-04-09T21:52:40.107 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12 2024-04-09T21:52:40.107 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg= 2024-04-09T21:52:40.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg=', 'Content-Length': '0'} 2024-04-09T21:52:40.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ddd6b284fd70314-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.109 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.109 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.109 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.109 DEBUG:boto:Path: /xryqqk-11/obj12 2024-04-09T21:52:40.109 DEBUG:boto:Data: 2024-04-09T21:52:40.109 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.110 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.110 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.110 DEBUG:boto:Params: {} 2024-04-09T21:52:40.110 DEBUG:boto:Token: None 2024-04-09T21:52:40.110 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12 2024-04-09T21:52:40.110 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg= 2024-04-09T21:52:40.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg=', 'Content-Length': '0'} 2024-04-09T21:52:40.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d415ae3a667b409-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj12 2024-04-09T21:52:40.112 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.112 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.112 DEBUG:boto:Method: GET 2024-04-09T21:52:40.113 DEBUG:boto:Path: /xryqqk-11/obj12 2024-04-09T21:52:40.113 DEBUG:boto:Data: 2024-04-09T21:52:40.113 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.113 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.113 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.113 DEBUG:boto:Params: {} 2024-04-09T21:52:40.113 DEBUG:boto:Token: None 2024-04-09T21:52:40.113 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12 2024-04-09T21:52:40.113 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ= 2024-04-09T21:52:40.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ=', 'Content-Length': '0'} 2024-04-09T21:52:40.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acf4bee53c711bca-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.115 DEBUG:boto:path=/xryqqk-11/obj12 2024-04-09T21:52:40.115 DEBUG:boto:auth_path=/xryqqk-11/obj12 2024-04-09T21:52:40.115 DEBUG:boto:Method: GET 2024-04-09T21:52:40.115 DEBUG:boto:Path: /xryqqk-11/obj12 2024-04-09T21:52:40.116 DEBUG:boto:Data: 2024-04-09T21:52:40.116 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.116 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.116 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.116 DEBUG:boto:Params: {} 2024-04-09T21:52:40.116 DEBUG:boto:Token: None 2024-04-09T21:52:40.116 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj12 2024-04-09T21:52:40.116 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ= 2024-04-09T21:52:40.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ=', 'Content-Length': '0'} 2024-04-09T21:52:40.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8d85d956671fca1-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.118 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.118 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.118 DEBUG:boto:path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.118 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.119 DEBUG:boto:Method: GET 2024-04-09T21:52:40.119 DEBUG:boto:Path: /xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.119 DEBUG:boto:Data: 2024-04-09T21:52:40.119 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.119 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.119 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.119 DEBUG:boto:Params: {} 2024-04-09T21:52:40.119 DEBUG:boto:Token: None 2024-04-09T21:52:40.119 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.119 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d80ff5e69f77c51-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.122 DEBUG:boto:b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.122 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.122 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.122 DEBUG:boto:path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.122 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.122 DEBUG:boto:Method: GET 2024-04-09T21:52:40.123 DEBUG:boto:Path: /xryqqk-11/?prefix=obj12 2024-04-09T21:52:40.123 DEBUG:boto:Data: 2024-04-09T21:52:40.123 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.123 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.123 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.123 DEBUG:boto:Params: {} 2024-04-09T21:52:40.123 DEBUG:boto:Token: None 2024-04-09T21:52:40.123 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.123 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b2357bf6e34857-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.126 DEBUG:boto:b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-04-09T21:52:40.126 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.126 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.126 DEBUG:boto:path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.126 DEBUG:boto:auth_path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.126 DEBUG:boto:Method: GET 2024-04-09T21:52:40.126 DEBUG:boto:Path: /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.127 DEBUG:boto:Data: 2024-04-09T21:52:40.127 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.127 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.127 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.127 DEBUG:boto:Params: {} 2024-04-09T21:52:40.127 DEBUG:boto:Token: None 2024-04-09T21:52:40.127 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.127 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-09T21:52:40.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-09T21:52:40.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0dd68974c8832d7-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.129 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.129 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.129 DEBUG:boto:path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.129 DEBUG:boto:auth_path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.130 DEBUG:boto:Method: GET 2024-04-09T21:52:40.130 DEBUG:boto:Path: /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.130 DEBUG:boto:Data: 2024-04-09T21:52:40.130 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.130 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.130 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.130 DEBUG:boto:Params: {} 2024-04-09T21:52:40.130 DEBUG:boto:Token: None 2024-04-09T21:52:40.130 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.130 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-09T21:52:40.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-09T21:52:40.132 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e330135cc5d1a968-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.132 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.132 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.132 DEBUG:boto:path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.132 DEBUG:boto:auth_path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.133 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.133 DEBUG:boto:Path: /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.133 DEBUG:boto:Data: 2024-04-09T21:52:40.133 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.133 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.133 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.133 DEBUG:boto:Params: {} 2024-04-09T21:52:40.133 DEBUG:boto:Token: None 2024-04-09T21:52:40.133 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.133 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk= 2024-04-09T21:52:40.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk=', 'Content-Length': '0'} 2024-04-09T21:52:40.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096c51ee1f3caf985-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.135 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.135 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.135 DEBUG:boto:path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.135 DEBUG:boto:auth_path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.135 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.136 DEBUG:boto:Path: /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.136 DEBUG:boto:Data: 2024-04-09T21:52:40.136 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.136 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.136 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.136 DEBUG:boto:Params: {} 2024-04-09T21:52:40.136 DEBUG:boto:Token: None 2024-04-09T21:52:40.136 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.136 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk= 2024-04-09T21:52:40.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk=', 'Content-Length': '0'} 2024-04-09T21:52:40.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006032a59e39656496-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-04-09T21:52:40.138 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.138 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.138 DEBUG:boto:path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.138 DEBUG:boto:auth_path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.139 DEBUG:boto:Method: GET 2024-04-09T21:52:40.139 DEBUG:boto:Path: /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.139 DEBUG:boto:Data: 2024-04-09T21:52:40.139 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.139 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.139 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.139 DEBUG:boto:Params: {} 2024-04-09T21:52:40.139 DEBUG:boto:Token: None 2024-04-09T21:52:40.139 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.139 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-09T21:52:40.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-09T21:52:40.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ecf076198a00c0d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.182 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.183 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.183 DEBUG:boto:path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.183 DEBUG:boto:auth_path=/xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.183 DEBUG:boto:Method: GET 2024-04-09T21:52:40.183 DEBUG:boto:Path: /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.183 DEBUG:boto:Data: 2024-04-09T21:52:40.183 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.183 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.183 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.183 DEBUG:boto:Params: {} 2024-04-09T21:52:40.183 DEBUG:boto:Token: None 2024-04-09T21:52:40.184 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13?versionId=null 2024-04-09T21:52:40.184 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-09T21:52:40.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-09T21:52:40.185 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c93d72be9d815ae-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.186 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.186 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.186 DEBUG:boto:path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.186 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.186 DEBUG:boto:Method: GET 2024-04-09T21:52:40.186 DEBUG:boto:Path: /xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.186 DEBUG:boto:Data: 2024-04-09T21:52:40.186 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.187 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.187 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.187 DEBUG:boto:Params: {} 2024-04-09T21:52:40.187 DEBUG:boto:Token: None 2024-04-09T21:52:40.187 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.187 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e415909ef9949c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.190 DEBUG:boto:b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.190 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.190 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.190 DEBUG:boto:path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.190 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.190 DEBUG:boto:Method: GET 2024-04-09T21:52:40.190 DEBUG:boto:Path: /xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.191 DEBUG:boto:Data: 2024-04-09T21:52:40.191 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.191 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.191 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.191 DEBUG:boto:Params: {} 2024-04-09T21:52:40.191 DEBUG:boto:Token: None 2024-04-09T21:52:40.191 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.191 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a48022451e95699-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.194 DEBUG:boto:b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.194 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.194 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.194 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.195 DEBUG:boto:Path: /xryqqk-11/obj13 2024-04-09T21:52:40.195 DEBUG:boto:Data: 2024-04-09T21:52:40.195 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.195 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.195 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.195 DEBUG:boto:Params: {} 2024-04-09T21:52:40.195 DEBUG:boto:Token: None 2024-04-09T21:52:40.195 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13 2024-04-09T21:52:40.195 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4= 2024-04-09T21:52:40.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4=', 'Content-Length': '0'} 2024-04-09T21:52:40.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002366d086586d8057-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.197 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.197 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.197 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.197 DEBUG:boto:Path: /xryqqk-11/obj13 2024-04-09T21:52:40.198 DEBUG:boto:Data: 2024-04-09T21:52:40.198 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.198 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.198 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.198 DEBUG:boto:Params: {} 2024-04-09T21:52:40.198 DEBUG:boto:Token: None 2024-04-09T21:52:40.198 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13 2024-04-09T21:52:40.198 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4= 2024-04-09T21:52:40.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4=', 'Content-Length': '0'} 2024-04-09T21:52:40.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b249ebb3dcfe0219-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj13 2024-04-09T21:52:40.200 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.200 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.201 DEBUG:boto:Method: GET 2024-04-09T21:52:40.201 DEBUG:boto:Path: /xryqqk-11/obj13 2024-04-09T21:52:40.201 DEBUG:boto:Data: 2024-04-09T21:52:40.201 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.201 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.201 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.201 DEBUG:boto:Params: {} 2024-04-09T21:52:40.201 DEBUG:boto:Token: None 2024-04-09T21:52:40.201 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13 2024-04-09T21:52:40.201 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY= 2024-04-09T21:52:40.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY=', 'Content-Length': '0'} 2024-04-09T21:52:40.203 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005229187ec76b6c95-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.203 DEBUG:boto:path=/xryqqk-11/obj13 2024-04-09T21:52:40.203 DEBUG:boto:auth_path=/xryqqk-11/obj13 2024-04-09T21:52:40.203 DEBUG:boto:Method: GET 2024-04-09T21:52:40.203 DEBUG:boto:Path: /xryqqk-11/obj13 2024-04-09T21:52:40.204 DEBUG:boto:Data: 2024-04-09T21:52:40.204 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.204 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.204 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.204 DEBUG:boto:Params: {} 2024-04-09T21:52:40.204 DEBUG:boto:Token: None 2024-04-09T21:52:40.204 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj13 2024-04-09T21:52:40.204 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY= 2024-04-09T21:52:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY=', 'Content-Length': '0'} 2024-04-09T21:52:40.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0fb002419a33921-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.206 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.206 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.206 DEBUG:boto:path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.206 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.206 DEBUG:boto:Method: GET 2024-04-09T21:52:40.206 DEBUG:boto:Path: /xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.207 DEBUG:boto:Data: 2024-04-09T21:52:40.207 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.207 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.207 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.207 DEBUG:boto:Params: {} 2024-04-09T21:52:40.207 DEBUG:boto:Token: None 2024-04-09T21:52:40.207 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.207 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548edc7b0c2009e2-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.209 DEBUG:boto:b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.210 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.210 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.210 DEBUG:boto:path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.210 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.210 DEBUG:boto:Method: GET 2024-04-09T21:52:40.210 DEBUG:boto:Path: /xryqqk-11/?prefix=obj13 2024-04-09T21:52:40.210 DEBUG:boto:Data: 2024-04-09T21:52:40.210 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.211 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.211 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.211 DEBUG:boto:Params: {} 2024-04-09T21:52:40.211 DEBUG:boto:Token: None 2024-04-09T21:52:40.211 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.211 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024834891bc18e97-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.213 DEBUG:boto:b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-04-09T21:52:40.214 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.214 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.214 DEBUG:boto:path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.214 DEBUG:boto:auth_path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.214 DEBUG:boto:Method: GET 2024-04-09T21:52:40.214 DEBUG:boto:Path: /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.214 DEBUG:boto:Data: 2024-04-09T21:52:40.214 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.214 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.215 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.215 DEBUG:boto:Params: {} 2024-04-09T21:52:40.215 DEBUG:boto:Token: None 2024-04-09T21:52:40.215 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.215 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-09T21:52:40.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-09T21:52:40.216 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d24d86d2f66a5a32-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.217 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.217 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.217 DEBUG:boto:path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.217 DEBUG:boto:auth_path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.217 DEBUG:boto:Method: GET 2024-04-09T21:52:40.217 DEBUG:boto:Path: /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.217 DEBUG:boto:Data: 2024-04-09T21:52:40.217 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.218 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.218 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.218 DEBUG:boto:Params: {} 2024-04-09T21:52:40.218 DEBUG:boto:Token: None 2024-04-09T21:52:40.218 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.218 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-09T21:52:40.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-09T21:52:40.220 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f153d5682873b1bc-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.220 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.220 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.220 DEBUG:boto:path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.220 DEBUG:boto:auth_path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.220 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.220 DEBUG:boto:Path: /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.221 DEBUG:boto:Data: 2024-04-09T21:52:40.221 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.221 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.221 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.221 DEBUG:boto:Params: {} 2024-04-09T21:52:40.221 DEBUG:boto:Token: None 2024-04-09T21:52:40.221 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.221 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU= 2024-04-09T21:52:40.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU=', 'Content-Length': '0'} 2024-04-09T21:52:40.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5c4e1eaca875691-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.223 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.223 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.223 DEBUG:boto:path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.223 DEBUG:boto:auth_path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.223 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.224 DEBUG:boto:Path: /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.224 DEBUG:boto:Data: 2024-04-09T21:52:40.224 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.224 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.224 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.224 DEBUG:boto:Params: {} 2024-04-09T21:52:40.224 DEBUG:boto:Token: None 2024-04-09T21:52:40.224 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.224 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU= 2024-04-09T21:52:40.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU=', 'Content-Length': '0'} 2024-04-09T21:52:40.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018784bb2a182d57e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-04-09T21:52:40.227 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.227 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.227 DEBUG:boto:path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.227 DEBUG:boto:auth_path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.227 DEBUG:boto:Method: GET 2024-04-09T21:52:40.227 DEBUG:boto:Path: /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.227 DEBUG:boto:Data: 2024-04-09T21:52:40.227 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.227 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.228 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.228 DEBUG:boto:Params: {} 2024-04-09T21:52:40.228 DEBUG:boto:Token: None 2024-04-09T21:52:40.228 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.228 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-09T21:52:40.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-09T21:52:40.230 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000deaaf179d6feb027-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.274 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.275 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.275 DEBUG:boto:path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.275 DEBUG:boto:auth_path=/xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.275 DEBUG:boto:Method: GET 2024-04-09T21:52:40.275 DEBUG:boto:Path: /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.275 DEBUG:boto:Data: 2024-04-09T21:52:40.275 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.275 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.275 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.275 DEBUG:boto:Params: {} 2024-04-09T21:52:40.275 DEBUG:boto:Token: None 2024-04-09T21:52:40.276 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14?versionId=null 2024-04-09T21:52:40.276 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-09T21:52:40.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-09T21:52:40.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a368af17ce02808-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.278 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.278 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.278 DEBUG:boto:path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.278 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.278 DEBUG:boto:Method: GET 2024-04-09T21:52:40.278 DEBUG:boto:Path: /xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.278 DEBUG:boto:Data: 2024-04-09T21:52:40.278 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.278 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.279 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.279 DEBUG:boto:Params: {} 2024-04-09T21:52:40.279 DEBUG:boto:Token: None 2024-04-09T21:52:40.279 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.279 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e052b589aad18e7a-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.281 DEBUG:boto:b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.282 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.282 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.282 DEBUG:boto:path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.282 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.282 DEBUG:boto:Method: GET 2024-04-09T21:52:40.282 DEBUG:boto:Path: /xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.282 DEBUG:boto:Data: 2024-04-09T21:52:40.282 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.282 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.283 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.283 DEBUG:boto:Params: {} 2024-04-09T21:52:40.283 DEBUG:boto:Token: None 2024-04-09T21:52:40.283 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.283 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca507c53ab0bbcf-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.285 DEBUG:boto:b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.285 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.286 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.286 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.286 DEBUG:boto:Path: /xryqqk-11/obj14 2024-04-09T21:52:40.286 DEBUG:boto:Data: 2024-04-09T21:52:40.286 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.286 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.286 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.286 DEBUG:boto:Params: {} 2024-04-09T21:52:40.286 DEBUG:boto:Token: None 2024-04-09T21:52:40.286 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14 2024-04-09T21:52:40.287 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo= 2024-04-09T21:52:40.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo=', 'Content-Length': '0'} 2024-04-09T21:52:40.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a166e64c4fb5dab-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.289 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.289 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.289 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.289 DEBUG:boto:Path: /xryqqk-11/obj14 2024-04-09T21:52:40.289 DEBUG:boto:Data: 2024-04-09T21:52:40.289 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.289 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.289 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.289 DEBUG:boto:Params: {} 2024-04-09T21:52:40.289 DEBUG:boto:Token: None 2024-04-09T21:52:40.290 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14 2024-04-09T21:52:40.290 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo= 2024-04-09T21:52:40.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo=', 'Content-Length': '0'} 2024-04-09T21:52:40.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000977ff40dc4e21cfd-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj14 2024-04-09T21:52:40.291 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.292 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.292 DEBUG:boto:Method: GET 2024-04-09T21:52:40.292 DEBUG:boto:Path: /xryqqk-11/obj14 2024-04-09T21:52:40.292 DEBUG:boto:Data: 2024-04-09T21:52:40.292 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.292 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.292 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.292 DEBUG:boto:Params: {} 2024-04-09T21:52:40.292 DEBUG:boto:Token: None 2024-04-09T21:52:40.292 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14 2024-04-09T21:52:40.292 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4= 2024-04-09T21:52:40.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4=', 'Content-Length': '0'} 2024-04-09T21:52:40.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93f983177471432-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.294 DEBUG:boto:path=/xryqqk-11/obj14 2024-04-09T21:52:40.294 DEBUG:boto:auth_path=/xryqqk-11/obj14 2024-04-09T21:52:40.295 DEBUG:boto:Method: GET 2024-04-09T21:52:40.295 DEBUG:boto:Path: /xryqqk-11/obj14 2024-04-09T21:52:40.295 DEBUG:boto:Data: 2024-04-09T21:52:40.295 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.295 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.295 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.295 DEBUG:boto:Params: {} 2024-04-09T21:52:40.295 DEBUG:boto:Token: None 2024-04-09T21:52:40.295 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj14 2024-04-09T21:52:40.295 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4= 2024-04-09T21:52:40.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4=', 'Content-Length': '0'} 2024-04-09T21:52:40.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c23a691d88afd2be-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.297 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.297 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.297 DEBUG:boto:path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.298 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.298 DEBUG:boto:Method: GET 2024-04-09T21:52:40.298 DEBUG:boto:Path: /xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.298 DEBUG:boto:Data: 2024-04-09T21:52:40.298 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.298 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.298 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.298 DEBUG:boto:Params: {} 2024-04-09T21:52:40.298 DEBUG:boto:Token: None 2024-04-09T21:52:40.298 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.299 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b31f9ef8f587d55-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.301 DEBUG:boto:b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.302 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.302 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.302 DEBUG:boto:path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.302 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.302 DEBUG:boto:Method: GET 2024-04-09T21:52:40.302 DEBUG:boto:Path: /xryqqk-11/?prefix=obj14 2024-04-09T21:52:40.302 DEBUG:boto:Data: 2024-04-09T21:52:40.302 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.302 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.302 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.302 DEBUG:boto:Params: {} 2024-04-09T21:52:40.303 DEBUG:boto:Token: None 2024-04-09T21:52:40.303 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.303 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091097cc8479b0bc5-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.305 DEBUG:boto:b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-04-09T21:52:40.305 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.306 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.306 DEBUG:boto:path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.306 DEBUG:boto:auth_path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.306 DEBUG:boto:Method: GET 2024-04-09T21:52:40.306 DEBUG:boto:Path: /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.306 DEBUG:boto:Data: 2024-04-09T21:52:40.306 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.307 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.307 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.307 DEBUG:boto:Params: {} 2024-04-09T21:52:40.307 DEBUG:boto:Token: None 2024-04-09T21:52:40.307 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.307 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-09T21:52:40.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-09T21:52:40.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf816f93b7862078-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.309 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.309 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.309 DEBUG:boto:path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.310 DEBUG:boto:auth_path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.310 DEBUG:boto:Method: GET 2024-04-09T21:52:40.310 DEBUG:boto:Path: /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.310 DEBUG:boto:Data: 2024-04-09T21:52:40.310 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.310 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.310 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.310 DEBUG:boto:Params: {} 2024-04-09T21:52:40.310 DEBUG:boto:Token: None 2024-04-09T21:52:40.310 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.311 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-09T21:52:40.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-09T21:52:40.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea7b164f4402a27c-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.313 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.313 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.313 DEBUG:boto:path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.313 DEBUG:boto:auth_path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.313 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.313 DEBUG:boto:Path: /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.313 DEBUG:boto:Data: 2024-04-09T21:52:40.313 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.313 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.313 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.314 DEBUG:boto:Params: {} 2024-04-09T21:52:40.314 DEBUG:boto:Token: None 2024-04-09T21:52:40.314 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.314 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y= 2024-04-09T21:52:40.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y=', 'Content-Length': '0'} 2024-04-09T21:52:40.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6e827b28b6f9015-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.316 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.316 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.316 DEBUG:boto:path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.316 DEBUG:boto:auth_path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.316 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.316 DEBUG:boto:Path: /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.316 DEBUG:boto:Data: 2024-04-09T21:52:40.316 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.316 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.316 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.316 DEBUG:boto:Params: {} 2024-04-09T21:52:40.317 DEBUG:boto:Token: None 2024-04-09T21:52:40.317 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.317 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y= 2024-04-09T21:52:40.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y=', 'Content-Length': '0'} 2024-04-09T21:52:40.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc262fec6d92ae0b-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-04-09T21:52:40.319 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.319 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.319 DEBUG:boto:path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.319 DEBUG:boto:auth_path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.320 DEBUG:boto:Method: GET 2024-04-09T21:52:40.320 DEBUG:boto:Path: /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.320 DEBUG:boto:Data: 2024-04-09T21:52:40.320 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.320 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.320 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.320 DEBUG:boto:Params: {} 2024-04-09T21:52:40.320 DEBUG:boto:Token: None 2024-04-09T21:52:40.320 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-09T21:52:40.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-09T21:52:40.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd509f4abad4e02e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.322 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.323 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.323 DEBUG:boto:path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.323 DEBUG:boto:auth_path=/xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.323 DEBUG:boto:Method: GET 2024-04-09T21:52:40.323 DEBUG:boto:Path: /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.323 DEBUG:boto:Data: 2024-04-09T21:52:40.323 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.323 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.323 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.323 DEBUG:boto:Params: {} 2024-04-09T21:52:40.323 DEBUG:boto:Token: None 2024-04-09T21:52:40.324 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15?versionId=null 2024-04-09T21:52:40.324 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-09T21:52:40.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-09T21:52:40.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c647ff5628e864-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.366 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.367 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.367 DEBUG:boto:path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.367 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.367 DEBUG:boto:Method: GET 2024-04-09T21:52:40.367 DEBUG:boto:Path: /xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.367 DEBUG:boto:Data: 2024-04-09T21:52:40.367 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.367 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.367 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.367 DEBUG:boto:Params: {} 2024-04-09T21:52:40.367 DEBUG:boto:Token: None 2024-04-09T21:52:40.368 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.368 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ef52b96e86941c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.370 DEBUG:boto:b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.370 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.371 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.371 DEBUG:boto:path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.371 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.371 DEBUG:boto:Method: GET 2024-04-09T21:52:40.371 DEBUG:boto:Path: /xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.371 DEBUG:boto:Data: 2024-04-09T21:52:40.371 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.371 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.371 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.371 DEBUG:boto:Params: {} 2024-04-09T21:52:40.371 DEBUG:boto:Token: None 2024-04-09T21:52:40.372 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.372 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005223fcfa28dc691d-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.374 DEBUG:boto:b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.374 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.374 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.375 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.375 DEBUG:boto:Path: /xryqqk-11/obj15 2024-04-09T21:52:40.375 DEBUG:boto:Data: 2024-04-09T21:52:40.375 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.375 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.375 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.375 DEBUG:boto:Params: {} 2024-04-09T21:52:40.375 DEBUG:boto:Token: None 2024-04-09T21:52:40.375 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15 2024-04-09T21:52:40.375 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs= 2024-04-09T21:52:40.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs=', 'Content-Length': '0'} 2024-04-09T21:52:40.377 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048087bbd29c6cc1d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.377 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.377 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.377 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.378 DEBUG:boto:Path: /xryqqk-11/obj15 2024-04-09T21:52:40.378 DEBUG:boto:Data: 2024-04-09T21:52:40.378 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.378 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.378 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.378 DEBUG:boto:Params: {} 2024-04-09T21:52:40.378 DEBUG:boto:Token: None 2024-04-09T21:52:40.378 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15 2024-04-09T21:52:40.378 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs= 2024-04-09T21:52:40.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs=', 'Content-Length': '0'} 2024-04-09T21:52:40.380 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399dddb77017bbf0-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj15 2024-04-09T21:52:40.380 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.381 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.381 DEBUG:boto:Method: GET 2024-04-09T21:52:40.381 DEBUG:boto:Path: /xryqqk-11/obj15 2024-04-09T21:52:40.381 DEBUG:boto:Data: 2024-04-09T21:52:40.381 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.381 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.381 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.381 DEBUG:boto:Params: {} 2024-04-09T21:52:40.381 DEBUG:boto:Token: None 2024-04-09T21:52:40.382 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15 2024-04-09T21:52:40.382 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic= 2024-04-09T21:52:40.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic=', 'Content-Length': '0'} 2024-04-09T21:52:40.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f6d5692c8e903d9-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.384 DEBUG:boto:path=/xryqqk-11/obj15 2024-04-09T21:52:40.384 DEBUG:boto:auth_path=/xryqqk-11/obj15 2024-04-09T21:52:40.384 DEBUG:boto:Method: GET 2024-04-09T21:52:40.384 DEBUG:boto:Path: /xryqqk-11/obj15 2024-04-09T21:52:40.384 DEBUG:boto:Data: 2024-04-09T21:52:40.384 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.384 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.384 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.384 DEBUG:boto:Params: {} 2024-04-09T21:52:40.385 DEBUG:boto:Token: None 2024-04-09T21:52:40.385 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj15 2024-04-09T21:52:40.385 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic= 2024-04-09T21:52:40.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic=', 'Content-Length': '0'} 2024-04-09T21:52:40.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0057df4b10d47ea-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.386 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.387 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.387 DEBUG:boto:path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.387 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.387 DEBUG:boto:Method: GET 2024-04-09T21:52:40.387 DEBUG:boto:Path: /xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.387 DEBUG:boto:Data: 2024-04-09T21:52:40.387 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.387 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.387 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.387 DEBUG:boto:Params: {} 2024-04-09T21:52:40.387 DEBUG:boto:Token: None 2024-04-09T21:52:40.388 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.388 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1aa45177c96c950-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.390 DEBUG:boto:b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.391 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.391 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.391 DEBUG:boto:path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.391 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.391 DEBUG:boto:Method: GET 2024-04-09T21:52:40.391 DEBUG:boto:Path: /xryqqk-11/?prefix=obj15 2024-04-09T21:52:40.391 DEBUG:boto:Data: 2024-04-09T21:52:40.391 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.391 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.391 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.391 DEBUG:boto:Params: {} 2024-04-09T21:52:40.392 DEBUG:boto:Token: None 2024-04-09T21:52:40.392 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.392 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008636d57fdb1c4a90-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.395 DEBUG:boto:b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-04-09T21:52:40.395 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.395 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.395 DEBUG:boto:path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.395 DEBUG:boto:auth_path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.395 DEBUG:boto:Method: GET 2024-04-09T21:52:40.395 DEBUG:boto:Path: /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.396 DEBUG:boto:Data: 2024-04-09T21:52:40.396 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.396 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.396 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.396 DEBUG:boto:Params: {} 2024-04-09T21:52:40.396 DEBUG:boto:Token: None 2024-04-09T21:52:40.396 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.396 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-09T21:52:40.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-09T21:52:40.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4f84ec8e01b6e83-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.398 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.398 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.398 DEBUG:boto:path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.399 DEBUG:boto:auth_path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.399 DEBUG:boto:Method: GET 2024-04-09T21:52:40.399 DEBUG:boto:Path: /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.399 DEBUG:boto:Data: 2024-04-09T21:52:40.399 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.399 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.399 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.399 DEBUG:boto:Params: {} 2024-04-09T21:52:40.399 DEBUG:boto:Token: None 2024-04-09T21:52:40.399 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.400 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-09T21:52:40.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-09T21:52:40.401 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fd86e8fb5246a69-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.446 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.446 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.447 DEBUG:boto:path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.447 DEBUG:boto:auth_path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.447 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.447 DEBUG:boto:Path: /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.447 DEBUG:boto:Data: 2024-04-09T21:52:40.447 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.447 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.447 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.447 DEBUG:boto:Params: {} 2024-04-09T21:52:40.447 DEBUG:boto:Token: None 2024-04-09T21:52:40.448 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.448 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk= 2024-04-09T21:52:40.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk=', 'Content-Length': '0'} 2024-04-09T21:52:40.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d946ba06ffa6ca61-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.450 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.450 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.450 DEBUG:boto:path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.450 DEBUG:boto:auth_path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.450 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.450 DEBUG:boto:Path: /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.450 DEBUG:boto:Data: 2024-04-09T21:52:40.451 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.451 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.451 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.451 DEBUG:boto:Params: {} 2024-04-09T21:52:40.451 DEBUG:boto:Token: None 2024-04-09T21:52:40.451 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.451 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk= 2024-04-09T21:52:40.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk=', 'Content-Length': '0'} 2024-04-09T21:52:40.453 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001274a5980674fa84-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-04-09T21:52:40.453 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.453 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.453 DEBUG:boto:path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.454 DEBUG:boto:auth_path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.454 DEBUG:boto:Method: GET 2024-04-09T21:52:40.454 DEBUG:boto:Path: /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.454 DEBUG:boto:Data: 2024-04-09T21:52:40.454 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.454 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.454 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.454 DEBUG:boto:Params: {} 2024-04-09T21:52:40.454 DEBUG:boto:Token: None 2024-04-09T21:52:40.454 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.455 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-09T21:52:40.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-09T21:52:40.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e140812f1ee7c2e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.456 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.456 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.457 DEBUG:boto:path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.457 DEBUG:boto:auth_path=/xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.457 DEBUG:boto:Method: GET 2024-04-09T21:52:40.457 DEBUG:boto:Path: /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.457 DEBUG:boto:Data: 2024-04-09T21:52:40.457 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.457 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.457 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.457 DEBUG:boto:Params: {} 2024-04-09T21:52:40.457 DEBUG:boto:Token: None 2024-04-09T21:52:40.457 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16?versionId=null 2024-04-09T21:52:40.458 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-09T21:52:40.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-09T21:52:40.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ac3e8917ca40ba9-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.459 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.460 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.460 DEBUG:boto:path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.460 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.460 DEBUG:boto:Method: GET 2024-04-09T21:52:40.460 DEBUG:boto:Path: /xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.460 DEBUG:boto:Data: 2024-04-09T21:52:40.460 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.460 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.460 DEBUG:boto:Params: {} 2024-04-09T21:52:40.461 DEBUG:boto:Token: None 2024-04-09T21:52:40.461 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.461 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb077f62e65e4f3e-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.463 DEBUG:boto:b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.464 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.464 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.464 DEBUG:boto:path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.464 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.464 DEBUG:boto:Method: GET 2024-04-09T21:52:40.464 DEBUG:boto:Path: /xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.464 DEBUG:boto:Data: 2024-04-09T21:52:40.464 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.464 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.465 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.465 DEBUG:boto:Params: {} 2024-04-09T21:52:40.465 DEBUG:boto:Token: None 2024-04-09T21:52:40.465 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.465 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54aec7ffeb14845-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.467 DEBUG:boto:b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.467 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.468 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.468 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.468 DEBUG:boto:Path: /xryqqk-11/obj16 2024-04-09T21:52:40.468 DEBUG:boto:Data: 2024-04-09T21:52:40.468 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.468 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.468 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.468 DEBUG:boto:Params: {} 2024-04-09T21:52:40.468 DEBUG:boto:Token: None 2024-04-09T21:52:40.468 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16 2024-04-09T21:52:40.469 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU= 2024-04-09T21:52:40.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU=', 'Content-Length': '0'} 2024-04-09T21:52:40.470 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a5a58a6997bd46d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.470 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.470 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.470 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.470 DEBUG:boto:Path: /xryqqk-11/obj16 2024-04-09T21:52:40.471 DEBUG:boto:Data: 2024-04-09T21:52:40.471 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.471 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.471 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.471 DEBUG:boto:Params: {} 2024-04-09T21:52:40.471 DEBUG:boto:Token: None 2024-04-09T21:52:40.471 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16 2024-04-09T21:52:40.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU= 2024-04-09T21:52:40.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU=', 'Content-Length': '0'} 2024-04-09T21:52:40.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f59cfb36d3f190f8-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj16 2024-04-09T21:52:40.473 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.473 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.473 DEBUG:boto:Method: GET 2024-04-09T21:52:40.473 DEBUG:boto:Path: /xryqqk-11/obj16 2024-04-09T21:52:40.473 DEBUG:boto:Data: 2024-04-09T21:52:40.473 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.473 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.474 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.474 DEBUG:boto:Params: {} 2024-04-09T21:52:40.474 DEBUG:boto:Token: None 2024-04-09T21:52:40.474 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16 2024-04-09T21:52:40.474 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM= 2024-04-09T21:52:40.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM=', 'Content-Length': '0'} 2024-04-09T21:52:40.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b350f6fc43d8667-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.476 DEBUG:boto:path=/xryqqk-11/obj16 2024-04-09T21:52:40.476 DEBUG:boto:auth_path=/xryqqk-11/obj16 2024-04-09T21:52:40.476 DEBUG:boto:Method: GET 2024-04-09T21:52:40.476 DEBUG:boto:Path: /xryqqk-11/obj16 2024-04-09T21:52:40.476 DEBUG:boto:Data: 2024-04-09T21:52:40.476 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.476 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.476 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.476 DEBUG:boto:Params: {} 2024-04-09T21:52:40.477 DEBUG:boto:Token: None 2024-04-09T21:52:40.477 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj16 2024-04-09T21:52:40.477 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM= 2024-04-09T21:52:40.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM=', 'Content-Length': '0'} 2024-04-09T21:52:40.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e98ce11499f4910-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.523 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.527 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.527 DEBUG:boto:path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.527 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.527 DEBUG:boto:Method: GET 2024-04-09T21:52:40.527 DEBUG:boto:Path: /xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.527 DEBUG:boto:Data: 2024-04-09T21:52:40.527 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.527 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.527 DEBUG:boto:Params: {} 2024-04-09T21:52:40.528 DEBUG:boto:Token: None 2024-04-09T21:52:40.528 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022aa14b50dc3422c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.530 DEBUG:boto:b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.531 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.531 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.531 DEBUG:boto:path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.531 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.531 DEBUG:boto:Method: GET 2024-04-09T21:52:40.531 DEBUG:boto:Path: /xryqqk-11/?prefix=obj16 2024-04-09T21:52:40.532 DEBUG:boto:Data: 2024-04-09T21:52:40.532 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.532 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.532 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.532 DEBUG:boto:Params: {} 2024-04-09T21:52:40.532 DEBUG:boto:Token: None 2024-04-09T21:52:40.532 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002610c14113bcca21-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.534 DEBUG:boto:b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-09T21:52:40.535 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.535 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.535 DEBUG:boto:path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.535 DEBUG:boto:auth_path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.535 DEBUG:boto:Method: GET 2024-04-09T21:52:40.535 DEBUG:boto:Path: /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.535 DEBUG:boto:Data: 2024-04-09T21:52:40.536 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.536 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.536 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.536 DEBUG:boto:Params: {} 2024-04-09T21:52:40.536 DEBUG:boto:Token: None 2024-04-09T21:52:40.536 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-09T21:52:40.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-09T21:52:40.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057865ff98ff5cafe-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.539 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.539 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.539 DEBUG:boto:path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.539 DEBUG:boto:auth_path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.540 DEBUG:boto:Method: GET 2024-04-09T21:52:40.540 DEBUG:boto:Path: /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.540 DEBUG:boto:Data: 2024-04-09T21:52:40.540 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.540 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.540 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.541 DEBUG:boto:Params: {} 2024-04-09T21:52:40.541 DEBUG:boto:Token: None 2024-04-09T21:52:40.541 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.541 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-09T21:52:40.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-09T21:52:40.543 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aa1717b6fbdaf39-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.544 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.544 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.544 DEBUG:boto:path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.544 DEBUG:boto:auth_path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.545 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.545 DEBUG:boto:Path: /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.545 DEBUG:boto:Data: 2024-04-09T21:52:40.545 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.545 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.546 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.546 DEBUG:boto:Params: {} 2024-04-09T21:52:40.546 DEBUG:boto:Token: None 2024-04-09T21:52:40.546 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.547 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA= 2024-04-09T21:52:40.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA=', 'Content-Length': '0'} 2024-04-09T21:52:40.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000673e479936a5af5e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.549 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.549 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.549 DEBUG:boto:path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.550 DEBUG:boto:auth_path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.550 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.550 DEBUG:boto:Path: /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.550 DEBUG:boto:Data: 2024-04-09T21:52:40.550 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.551 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.551 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.551 DEBUG:boto:Params: {} 2024-04-09T21:52:40.551 DEBUG:boto:Token: None 2024-04-09T21:52:40.552 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.552 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA= 2024-04-09T21:52:40.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA=', 'Content-Length': '0'} 2024-04-09T21:52:40.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034113c91defaf0e8-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-09T21:52:40.555 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.555 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.555 DEBUG:boto:path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.555 DEBUG:boto:auth_path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.555 DEBUG:boto:Method: GET 2024-04-09T21:52:40.555 DEBUG:boto:Path: /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.556 DEBUG:boto:Data: 2024-04-09T21:52:40.556 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.556 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.556 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.556 DEBUG:boto:Params: {} 2024-04-09T21:52:40.557 DEBUG:boto:Token: None 2024-04-09T21:52:40.557 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.557 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-09T21:52:40.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-09T21:52:40.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ca8bb9600c6572e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.602 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.603 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.603 DEBUG:boto:path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.603 DEBUG:boto:auth_path=/xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.603 DEBUG:boto:Method: GET 2024-04-09T21:52:40.603 DEBUG:boto:Path: /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.603 DEBUG:boto:Data: 2024-04-09T21:52:40.603 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.603 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.603 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.603 DEBUG:boto:Params: {} 2024-04-09T21:52:40.603 DEBUG:boto:Token: None 2024-04-09T21:52:40.604 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17?versionId=null 2024-04-09T21:52:40.604 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-09T21:52:40.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-09T21:52:40.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030ec3068010beb8a-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.606 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.606 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.606 DEBUG:boto:path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.606 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.607 DEBUG:boto:Method: GET 2024-04-09T21:52:40.607 DEBUG:boto:Path: /xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.607 DEBUG:boto:Data: 2024-04-09T21:52:40.607 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.607 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.607 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.607 DEBUG:boto:Params: {} 2024-04-09T21:52:40.607 DEBUG:boto:Token: None 2024-04-09T21:52:40.608 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.608 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac1d8ab2dea2a3c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.610 DEBUG:boto:b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.610 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.610 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.611 DEBUG:boto:path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.611 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.611 DEBUG:boto:Method: GET 2024-04-09T21:52:40.611 DEBUG:boto:Path: /xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.611 DEBUG:boto:Data: 2024-04-09T21:52:40.611 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.611 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.611 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.611 DEBUG:boto:Params: {} 2024-04-09T21:52:40.611 DEBUG:boto:Token: None 2024-04-09T21:52:40.611 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.612 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b460ea41bc667b-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.614 DEBUG:boto:b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.614 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.614 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.614 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.614 DEBUG:boto:Path: /xryqqk-11/obj17 2024-04-09T21:52:40.614 DEBUG:boto:Data: 2024-04-09T21:52:40.615 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.615 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.615 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.615 DEBUG:boto:Params: {} 2024-04-09T21:52:40.615 DEBUG:boto:Token: None 2024-04-09T21:52:40.615 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17 2024-04-09T21:52:40.615 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w= 2024-04-09T21:52:40.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w=', 'Content-Length': '0'} 2024-04-09T21:52:40.617 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe027e6e336eee20-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.617 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.617 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.618 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.618 DEBUG:boto:Path: /xryqqk-11/obj17 2024-04-09T21:52:40.618 DEBUG:boto:Data: 2024-04-09T21:52:40.618 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.618 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.618 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.618 DEBUG:boto:Params: {} 2024-04-09T21:52:40.618 DEBUG:boto:Token: None 2024-04-09T21:52:40.618 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17 2024-04-09T21:52:40.619 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w= 2024-04-09T21:52:40.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w=', 'Content-Length': '0'} 2024-04-09T21:52:40.620 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba3b4c112d13bdc6-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-09T21:52:40.621 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.621 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.621 DEBUG:boto:Method: GET 2024-04-09T21:52:40.621 DEBUG:boto:Path: /xryqqk-11/obj17 2024-04-09T21:52:40.621 DEBUG:boto:Data: 2024-04-09T21:52:40.621 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.621 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.621 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.621 DEBUG:boto:Params: {} 2024-04-09T21:52:40.622 DEBUG:boto:Token: None 2024-04-09T21:52:40.622 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17 2024-04-09T21:52:40.622 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds= 2024-04-09T21:52:40.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds=', 'Content-Length': '0'} 2024-04-09T21:52:40.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c6cc276bef32825-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.624 DEBUG:boto:path=/xryqqk-11/obj17 2024-04-09T21:52:40.624 DEBUG:boto:auth_path=/xryqqk-11/obj17 2024-04-09T21:52:40.624 DEBUG:boto:Method: GET 2024-04-09T21:52:40.624 DEBUG:boto:Path: /xryqqk-11/obj17 2024-04-09T21:52:40.625 DEBUG:boto:Data: 2024-04-09T21:52:40.625 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.625 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.625 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.625 DEBUG:boto:Params: {} 2024-04-09T21:52:40.625 DEBUG:boto:Token: None 2024-04-09T21:52:40.625 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj17 2024-04-09T21:52:40.625 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds= 2024-04-09T21:52:40.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds=', 'Content-Length': '0'} 2024-04-09T21:52:40.627 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2538920ab53f896-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.627 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.627 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.627 DEBUG:boto:path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.628 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.628 DEBUG:boto:Method: GET 2024-04-09T21:52:40.628 DEBUG:boto:Path: /xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.628 DEBUG:boto:Data: 2024-04-09T21:52:40.628 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.628 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.628 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.628 DEBUG:boto:Params: {} 2024-04-09T21:52:40.628 DEBUG:boto:Token: None 2024-04-09T21:52:40.628 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.628 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3626d59786bd1af-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.631 DEBUG:boto:b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.631 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.631 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.631 DEBUG:boto:path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.631 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.631 DEBUG:boto:Method: GET 2024-04-09T21:52:40.632 DEBUG:boto:Path: /xryqqk-11/?prefix=obj17 2024-04-09T21:52:40.632 DEBUG:boto:Data: 2024-04-09T21:52:40.632 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.632 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.632 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.632 DEBUG:boto:Params: {} 2024-04-09T21:52:40.632 DEBUG:boto:Token: None 2024-04-09T21:52:40.632 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.632 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a08d7cb0cfb70e64-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.678 DEBUG:boto:b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-04-09T21:52:40.679 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.679 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.679 DEBUG:boto:path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.679 DEBUG:boto:auth_path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.679 DEBUG:boto:Method: GET 2024-04-09T21:52:40.679 DEBUG:boto:Path: /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.680 DEBUG:boto:Data: 2024-04-09T21:52:40.680 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.680 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.680 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.680 DEBUG:boto:Params: {} 2024-04-09T21:52:40.680 DEBUG:boto:Token: None 2024-04-09T21:52:40.680 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.681 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-09T21:52:40.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-09T21:52:40.682 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dba64bc996ddb82f-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.683 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.683 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.683 DEBUG:boto:path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.683 DEBUG:boto:auth_path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.683 DEBUG:boto:Method: GET 2024-04-09T21:52:40.683 DEBUG:boto:Path: /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.683 DEBUG:boto:Data: 2024-04-09T21:52:40.683 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.683 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.684 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.684 DEBUG:boto:Params: {} 2024-04-09T21:52:40.684 DEBUG:boto:Token: None 2024-04-09T21:52:40.684 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.684 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-09T21:52:40.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-09T21:52:40.686 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019cdee40d6727de8-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.686 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.686 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.687 DEBUG:boto:path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.687 DEBUG:boto:auth_path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.687 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.687 DEBUG:boto:Path: /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.687 DEBUG:boto:Data: 2024-04-09T21:52:40.687 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.687 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.687 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.687 DEBUG:boto:Params: {} 2024-04-09T21:52:40.687 DEBUG:boto:Token: None 2024-04-09T21:52:40.688 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.688 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU= 2024-04-09T21:52:40.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU=', 'Content-Length': '0'} 2024-04-09T21:52:40.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0a50a6c040d0999-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.689 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.690 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.690 DEBUG:boto:path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.690 DEBUG:boto:auth_path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.690 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.690 DEBUG:boto:Path: /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.690 DEBUG:boto:Data: 2024-04-09T21:52:40.690 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.690 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.690 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.690 DEBUG:boto:Params: {} 2024-04-09T21:52:40.691 DEBUG:boto:Token: None 2024-04-09T21:52:40.691 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.691 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU= 2024-04-09T21:52:40.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU=', 'Content-Length': '0'} 2024-04-09T21:52:40.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048380b1a0de2d430-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-04-09T21:52:40.693 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.693 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.693 DEBUG:boto:path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.693 DEBUG:boto:auth_path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.693 DEBUG:boto:Method: GET 2024-04-09T21:52:40.693 DEBUG:boto:Path: /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.693 DEBUG:boto:Data: 2024-04-09T21:52:40.694 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.694 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.694 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.694 DEBUG:boto:Params: {} 2024-04-09T21:52:40.694 DEBUG:boto:Token: None 2024-04-09T21:52:40.694 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.694 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-09T21:52:40.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-09T21:52:40.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab91b8ea29c1eed0-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.696 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.696 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.697 DEBUG:boto:path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.697 DEBUG:boto:auth_path=/xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.697 DEBUG:boto:Method: GET 2024-04-09T21:52:40.697 DEBUG:boto:Path: /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.697 DEBUG:boto:Data: 2024-04-09T21:52:40.697 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.697 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.697 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.697 DEBUG:boto:Params: {} 2024-04-09T21:52:40.697 DEBUG:boto:Token: None 2024-04-09T21:52:40.697 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18?versionId=null 2024-04-09T21:52:40.698 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-09T21:52:40.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-09T21:52:40.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bcf85b13b67de94-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.700 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.700 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.700 DEBUG:boto:path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.700 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.700 DEBUG:boto:Method: GET 2024-04-09T21:52:40.700 DEBUG:boto:Path: /xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.700 DEBUG:boto:Data: 2024-04-09T21:52:40.700 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.700 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.701 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.701 DEBUG:boto:Params: {} 2024-04-09T21:52:40.701 DEBUG:boto:Token: None 2024-04-09T21:52:40.701 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.701 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc415627da04629-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.703 DEBUG:boto:b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.704 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.704 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.704 DEBUG:boto:path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.704 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.704 DEBUG:boto:Method: GET 2024-04-09T21:52:40.704 DEBUG:boto:Path: /xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.705 DEBUG:boto:Data: 2024-04-09T21:52:40.705 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.705 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.705 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.705 DEBUG:boto:Params: {} 2024-04-09T21:52:40.705 DEBUG:boto:Token: None 2024-04-09T21:52:40.705 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.705 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f8c88d608b6a05-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.708 DEBUG:boto:b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.708 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.708 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.708 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.708 DEBUG:boto:Path: /xryqqk-11/obj18 2024-04-09T21:52:40.708 DEBUG:boto:Data: 2024-04-09T21:52:40.709 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.709 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.709 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.709 DEBUG:boto:Params: {} 2024-04-09T21:52:40.709 DEBUG:boto:Token: None 2024-04-09T21:52:40.709 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18 2024-04-09T21:52:40.709 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI= 2024-04-09T21:52:40.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI=', 'Content-Length': '0'} 2024-04-09T21:52:40.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004f4cb35b2bf64b2-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.711 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.711 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.711 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.711 DEBUG:boto:Path: /xryqqk-11/obj18 2024-04-09T21:52:40.711 DEBUG:boto:Data: 2024-04-09T21:52:40.711 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.712 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.712 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.712 DEBUG:boto:Params: {} 2024-04-09T21:52:40.712 DEBUG:boto:Token: None 2024-04-09T21:52:40.712 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18 2024-04-09T21:52:40.712 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI= 2024-04-09T21:52:40.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI=', 'Content-Length': '0'} 2024-04-09T21:52:40.714 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6e51096f5b6c907-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj18 2024-04-09T21:52:40.714 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.714 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.714 DEBUG:boto:Method: GET 2024-04-09T21:52:40.715 DEBUG:boto:Path: /xryqqk-11/obj18 2024-04-09T21:52:40.715 DEBUG:boto:Data: 2024-04-09T21:52:40.715 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.715 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.715 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.715 DEBUG:boto:Params: {} 2024-04-09T21:52:40.715 DEBUG:boto:Token: None 2024-04-09T21:52:40.715 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18 2024-04-09T21:52:40.715 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk= 2024-04-09T21:52:40.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk=', 'Content-Length': '0'} 2024-04-09T21:52:40.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095755bee41fa034a-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.762 DEBUG:boto:path=/xryqqk-11/obj18 2024-04-09T21:52:40.763 DEBUG:boto:auth_path=/xryqqk-11/obj18 2024-04-09T21:52:40.763 DEBUG:boto:Method: GET 2024-04-09T21:52:40.763 DEBUG:boto:Path: /xryqqk-11/obj18 2024-04-09T21:52:40.763 DEBUG:boto:Data: 2024-04-09T21:52:40.763 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.763 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.763 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.763 DEBUG:boto:Params: {} 2024-04-09T21:52:40.763 DEBUG:boto:Token: None 2024-04-09T21:52:40.764 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj18 2024-04-09T21:52:40.764 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk= 2024-04-09T21:52:40.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk=', 'Content-Length': '0'} 2024-04-09T21:52:40.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002c03a50a968513d-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.766 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.766 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.766 DEBUG:boto:path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.766 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.766 DEBUG:boto:Method: GET 2024-04-09T21:52:40.766 DEBUG:boto:Path: /xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.766 DEBUG:boto:Data: 2024-04-09T21:52:40.766 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.766 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.767 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.767 DEBUG:boto:Params: {} 2024-04-09T21:52:40.767 DEBUG:boto:Token: None 2024-04-09T21:52:40.767 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.767 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776709abbdbd9270-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.769 DEBUG:boto:b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.770 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.770 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.770 DEBUG:boto:path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.770 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.770 DEBUG:boto:Method: GET 2024-04-09T21:52:40.770 DEBUG:boto:Path: /xryqqk-11/?prefix=obj18 2024-04-09T21:52:40.770 DEBUG:boto:Data: 2024-04-09T21:52:40.770 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.771 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.771 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.771 DEBUG:boto:Params: {} 2024-04-09T21:52:40.771 DEBUG:boto:Token: None 2024-04-09T21:52:40.771 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.771 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066785bdbb5cd914c-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.774 DEBUG:boto:b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-09T21:52:40.774 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.774 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.774 DEBUG:boto:path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.774 DEBUG:boto:auth_path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.774 DEBUG:boto:Method: GET 2024-04-09T21:52:40.775 DEBUG:boto:Path: /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.775 DEBUG:boto:Data: 2024-04-09T21:52:40.775 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.775 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.775 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.775 DEBUG:boto:Params: {} 2024-04-09T21:52:40.775 DEBUG:boto:Token: None 2024-04-09T21:52:40.775 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.775 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-09T21:52:40.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-09T21:52:40.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000087025e1d3cda1fc-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.777 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.777 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.777 DEBUG:boto:path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.778 DEBUG:boto:auth_path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.778 DEBUG:boto:Method: GET 2024-04-09T21:52:40.778 DEBUG:boto:Path: /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.778 DEBUG:boto:Data: 2024-04-09T21:52:40.778 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.778 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.778 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.778 DEBUG:boto:Params: {} 2024-04-09T21:52:40.778 DEBUG:boto:Token: None 2024-04-09T21:52:40.778 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.779 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-09T21:52:40.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-09T21:52:40.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c2de69d02495bb6-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.781 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.781 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.781 DEBUG:boto:path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.781 DEBUG:boto:auth_path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.781 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.781 DEBUG:boto:Path: /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.781 DEBUG:boto:Data: 2024-04-09T21:52:40.781 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.781 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.781 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.781 DEBUG:boto:Params: {} 2024-04-09T21:52:40.782 DEBUG:boto:Token: None 2024-04-09T21:52:40.782 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.782 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o= 2024-04-09T21:52:40.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o=', 'Content-Length': '0'} 2024-04-09T21:52:40.784 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca3b7b85e56aa6a9-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.784 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.784 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.784 DEBUG:boto:path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.784 DEBUG:boto:auth_path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.784 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.784 DEBUG:boto:Path: /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.784 DEBUG:boto:Data: 2024-04-09T21:52:40.784 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.785 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.785 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.785 DEBUG:boto:Params: {} 2024-04-09T21:52:40.785 DEBUG:boto:Token: None 2024-04-09T21:52:40.785 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.785 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o= 2024-04-09T21:52:40.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o=', 'Content-Length': '0'} 2024-04-09T21:52:40.787 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001654feff730ecce9-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-09T21:52:40.787 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.788 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.788 DEBUG:boto:path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.788 DEBUG:boto:auth_path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.788 DEBUG:boto:Method: GET 2024-04-09T21:52:40.788 DEBUG:boto:Path: /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.788 DEBUG:boto:Data: 2024-04-09T21:52:40.788 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.788 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.788 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.788 DEBUG:boto:Params: {} 2024-04-09T21:52:40.789 DEBUG:boto:Token: None 2024-04-09T21:52:40.789 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.789 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-09T21:52:40.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-09T21:52:40.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2b40b0c69b5adab-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.791 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.791 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.791 DEBUG:boto:path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.791 DEBUG:boto:auth_path=/xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.791 DEBUG:boto:Method: GET 2024-04-09T21:52:40.791 DEBUG:boto:Path: /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.791 DEBUG:boto:Data: 2024-04-09T21:52:40.791 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.792 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.792 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.792 DEBUG:boto:Params: {} 2024-04-09T21:52:40.792 DEBUG:boto:Token: None 2024-04-09T21:52:40.792 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19?versionId=null 2024-04-09T21:52:40.792 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-09T21:52:40.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-09T21:52:40.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006800cf467a4eb0cd-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.834 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.835 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.835 DEBUG:boto:path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.835 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.835 DEBUG:boto:Method: GET 2024-04-09T21:52:40.835 DEBUG:boto:Path: /xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.835 DEBUG:boto:Data: 2024-04-09T21:52:40.835 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.835 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.835 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.835 DEBUG:boto:Params: {} 2024-04-09T21:52:40.836 DEBUG:boto:Token: None 2024-04-09T21:52:40.836 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.836 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e785dd966ebf66-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.838 DEBUG:boto:b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.839 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.839 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.839 DEBUG:boto:path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.839 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.839 DEBUG:boto:Method: GET 2024-04-09T21:52:40.840 DEBUG:boto:Path: /xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.840 DEBUG:boto:Data: 2024-04-09T21:52:40.840 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.840 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.840 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.840 DEBUG:boto:Params: {} 2024-04-09T21:52:40.840 DEBUG:boto:Token: None 2024-04-09T21:52:40.840 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.840 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c050642cba8872b-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.843 DEBUG:boto:b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.843 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.843 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.843 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.844 DEBUG:boto:Path: /xryqqk-11/obj19 2024-04-09T21:52:40.844 DEBUG:boto:Data: 2024-04-09T21:52:40.844 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.844 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.844 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.844 DEBUG:boto:Params: {} 2024-04-09T21:52:40.844 DEBUG:boto:Token: None 2024-04-09T21:52:40.844 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19 2024-04-09T21:52:40.844 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ= 2024-04-09T21:52:40.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ=', 'Content-Length': '0'} 2024-04-09T21:52:40.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb9d161b9dbc8c52-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.846 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.847 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.847 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.847 DEBUG:boto:Path: /xryqqk-11/obj19 2024-04-09T21:52:40.847 DEBUG:boto:Data: 2024-04-09T21:52:40.847 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.847 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.847 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.847 DEBUG:boto:Params: {} 2024-04-09T21:52:40.847 DEBUG:boto:Token: None 2024-04-09T21:52:40.847 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19 2024-04-09T21:52:40.848 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ= 2024-04-09T21:52:40.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ=', 'Content-Length': '0'} 2024-04-09T21:52:40.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e1c636d3b1833e0-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-09T21:52:40.849 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.850 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.850 DEBUG:boto:Method: GET 2024-04-09T21:52:40.850 DEBUG:boto:Path: /xryqqk-11/obj19 2024-04-09T21:52:40.850 DEBUG:boto:Data: 2024-04-09T21:52:40.850 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.850 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.850 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.851 DEBUG:boto:Params: {} 2024-04-09T21:52:40.851 DEBUG:boto:Token: None 2024-04-09T21:52:40.851 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19 2024-04-09T21:52:40.851 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8= 2024-04-09T21:52:40.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8=', 'Content-Length': '0'} 2024-04-09T21:52:40.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e427d0e4537f8f9-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.853 DEBUG:boto:path=/xryqqk-11/obj19 2024-04-09T21:52:40.853 DEBUG:boto:auth_path=/xryqqk-11/obj19 2024-04-09T21:52:40.853 DEBUG:boto:Method: GET 2024-04-09T21:52:40.853 DEBUG:boto:Path: /xryqqk-11/obj19 2024-04-09T21:52:40.853 DEBUG:boto:Data: 2024-04-09T21:52:40.853 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.853 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.853 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.853 DEBUG:boto:Params: {} 2024-04-09T21:52:40.854 DEBUG:boto:Token: None 2024-04-09T21:52:40.854 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj19 2024-04-09T21:52:40.854 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8= 2024-04-09T21:52:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8=', 'Content-Length': '0'} 2024-04-09T21:52:40.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fff979bb276bfb66-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.856 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.856 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.856 DEBUG:boto:path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.856 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.856 DEBUG:boto:Method: GET 2024-04-09T21:52:40.856 DEBUG:boto:Path: /xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.856 DEBUG:boto:Data: 2024-04-09T21:52:40.856 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.857 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.857 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.857 DEBUG:boto:Params: {} 2024-04-09T21:52:40.857 DEBUG:boto:Token: None 2024-04-09T21:52:40.857 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.857 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e66bfd5ddde56b1-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.859 DEBUG:boto:b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.860 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.860 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.860 DEBUG:boto:path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.860 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.860 DEBUG:boto:Method: GET 2024-04-09T21:52:40.860 DEBUG:boto:Path: /xryqqk-11/?prefix=obj19 2024-04-09T21:52:40.860 DEBUG:boto:Data: 2024-04-09T21:52:40.861 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.861 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.861 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.861 DEBUG:boto:Params: {} 2024-04-09T21:52:40.861 DEBUG:boto:Token: None 2024-04-09T21:52:40.861 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.861 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc31e23d6d34b319-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.864 DEBUG:boto:b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-09T21:52:40.864 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.864 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.864 DEBUG:boto:path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.864 DEBUG:boto:auth_path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.865 DEBUG:boto:Method: GET 2024-04-09T21:52:40.865 DEBUG:boto:Path: /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.865 DEBUG:boto:Data: 2024-04-09T21:52:40.865 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.865 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.865 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.865 DEBUG:boto:Params: {} 2024-04-09T21:52:40.865 DEBUG:boto:Token: None 2024-04-09T21:52:40.865 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.865 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-09T21:52:40.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-09T21:52:40.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096784bd98d989f41-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.914 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.915 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.915 DEBUG:boto:path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.915 DEBUG:boto:auth_path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.916 DEBUG:boto:Method: GET 2024-04-09T21:52:40.916 DEBUG:boto:Path: /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.916 DEBUG:boto:Data: 2024-04-09T21:52:40.916 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.916 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.916 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.917 DEBUG:boto:Params: {} 2024-04-09T21:52:40.917 DEBUG:boto:Token: None 2024-04-09T21:52:40.917 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.918 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-09T21:52:40.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-09T21:52:40.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000463cfb3e0232a890-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.920 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.920 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.921 DEBUG:boto:path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.921 DEBUG:boto:auth_path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.921 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.921 DEBUG:boto:Path: /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.921 DEBUG:boto:Data: 2024-04-09T21:52:40.922 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.922 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.922 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.922 DEBUG:boto:Params: {} 2024-04-09T21:52:40.922 DEBUG:boto:Token: None 2024-04-09T21:52:40.923 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.923 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8= 2024-04-09T21:52:40.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8=', 'Content-Length': '0'} 2024-04-09T21:52:40.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000873515efbf943bf5-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.925 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.926 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.926 DEBUG:boto:path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.926 DEBUG:boto:auth_path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.926 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.926 DEBUG:boto:Path: /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.927 DEBUG:boto:Data: 2024-04-09T21:52:40.927 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.927 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.927 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.927 DEBUG:boto:Params: {} 2024-04-09T21:52:40.928 DEBUG:boto:Token: None 2024-04-09T21:52:40.928 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.928 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8= 2024-04-09T21:52:40.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8=', 'Content-Length': '0'} 2024-04-09T21:52:40.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2d899746930f5e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-09T21:52:40.931 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.931 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.931 DEBUG:boto:path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.932 DEBUG:boto:auth_path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.932 DEBUG:boto:Method: GET 2024-04-09T21:52:40.932 DEBUG:boto:Path: /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.932 DEBUG:boto:Data: 2024-04-09T21:52:40.932 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.932 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.933 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.933 DEBUG:boto:Params: {} 2024-04-09T21:52:40.933 DEBUG:boto:Token: None 2024-04-09T21:52:40.933 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.934 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-09T21:52:40.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-09T21:52:40.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c7f912f642c9b4b-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.936 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.936 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.936 DEBUG:boto:path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.936 DEBUG:boto:auth_path=/xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.936 DEBUG:boto:Method: GET 2024-04-09T21:52:40.936 DEBUG:boto:Path: /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.937 DEBUG:boto:Data: 2024-04-09T21:52:40.937 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.937 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.937 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.937 DEBUG:boto:Params: {} 2024-04-09T21:52:40.937 DEBUG:boto:Token: None 2024-04-09T21:52:40.937 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2?versionId=null 2024-04-09T21:52:40.938 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-09T21:52:40.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-09T21:52:40.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093395228e39f1d68-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.939 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.939 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.939 DEBUG:boto:path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:40.940 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:40.940 DEBUG:boto:Method: GET 2024-04-09T21:52:40.940 DEBUG:boto:Path: /xryqqk-11/?prefix=obj2 2024-04-09T21:52:40.940 DEBUG:boto:Data: 2024-04-09T21:52:40.940 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.940 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.940 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.940 DEBUG:boto:Params: {} 2024-04-09T21:52:40.940 DEBUG:boto:Token: None 2024-04-09T21:52:40.940 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.941 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052af4de31b890806-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.943 DEBUG:boto:b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.944 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:40.944 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:40.944 DEBUG:boto:path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:40.944 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:40.944 DEBUG:boto:Method: GET 2024-04-09T21:52:40.944 DEBUG:boto:Path: /xryqqk-11/?prefix=obj2 2024-04-09T21:52:40.944 DEBUG:boto:Data: 2024-04-09T21:52:40.945 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.945 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.945 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.945 DEBUG:boto:Params: {} 2024-04-09T21:52:40.945 DEBUG:boto:Token: None 2024-04-09T21:52:40.945 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/ 2024-04-09T21:52:40.945 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-09T21:52:40.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-09T21:52:40.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd94ea1700d83f4c-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.948 DEBUG:boto:b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:40.949 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.950 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.950 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.950 DEBUG:boto:Path: /xryqqk-11/obj2 2024-04-09T21:52:40.950 DEBUG:boto:Data: 2024-04-09T21:52:40.950 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.950 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.951 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.951 DEBUG:boto:Params: {} 2024-04-09T21:52:40.951 DEBUG:boto:Token: None 2024-04-09T21:52:40.951 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2 2024-04-09T21:52:40.951 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk= 2024-04-09T21:52:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk=', 'Content-Length': '0'} 2024-04-09T21:52:40.953 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8679fe634c9a469-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.954 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.954 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.954 DEBUG:boto:Method: HEAD 2024-04-09T21:52:40.954 DEBUG:boto:Path: /xryqqk-11/obj2 2024-04-09T21:52:40.955 DEBUG:boto:Data: 2024-04-09T21:52:40.955 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.955 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.955 DEBUG:boto:Params: {} 2024-04-09T21:52:40.956 DEBUG:boto:Token: None 2024-04-09T21:52:40.956 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2 2024-04-09T21:52:40.956 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk= 2024-04-09T21:52:40.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk=', 'Content-Length': '0'} 2024-04-09T21:52:40.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000445cdae5c6dc038e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-09T21:52:40.959 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.959 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.959 DEBUG:boto:Method: GET 2024-04-09T21:52:40.959 DEBUG:boto:Path: /xryqqk-11/obj2 2024-04-09T21:52:40.960 DEBUG:boto:Data: 2024-04-09T21:52:40.960 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.960 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:40.960 DEBUG:boto:Port: 8001 2024-04-09T21:52:40.960 DEBUG:boto:Params: {} 2024-04-09T21:52:40.961 DEBUG:boto:Token: None 2024-04-09T21:52:40.961 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2 2024-04-09T21:52:40.961 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo= 2024-04-09T21:52:40.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo=', 'Content-Length': '0'} 2024-04-09T21:52:40.963 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b04402f3c139c10-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:40.964 DEBUG:boto:path=/xryqqk-11/obj2 2024-04-09T21:52:40.964 DEBUG:boto:auth_path=/xryqqk-11/obj2 2024-04-09T21:52:40.964 DEBUG:boto:Method: GET 2024-04-09T21:52:40.965 DEBUG:boto:Path: /xryqqk-11/obj2 2024-04-09T21:52:40.965 DEBUG:boto:Data: 2024-04-09T21:52:40.965 DEBUG:boto:Headers: {} 2024-04-09T21:52:40.965 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:40.965 DEBUG:boto:Port: 8000 2024-04-09T21:52:40.965 DEBUG:boto:Params: {} 2024-04-09T21:52:40.966 DEBUG:boto:Token: None 2024-04-09T21:52:40.966 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:40 GMT /xryqqk-11/obj2 2024-04-09T21:52:40.966 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo= 2024-04-09T21:52:40.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo=', 'Content-Length': '0'} 2024-04-09T21:52:40.968 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001cdb091c6cfc20e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.010 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.011 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.011 DEBUG:boto:path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:41.011 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:41.011 DEBUG:boto:Method: GET 2024-04-09T21:52:41.012 DEBUG:boto:Path: /xryqqk-11/?prefix=obj2 2024-04-09T21:52:41.012 DEBUG:boto:Data: 2024-04-09T21:52:41.012 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.012 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.012 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.013 DEBUG:boto:Params: {} 2024-04-09T21:52:41.013 DEBUG:boto:Token: None 2024-04-09T21:52:41.013 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.013 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07f46eaa6174183-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.017 DEBUG:boto:b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.018 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.018 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.018 DEBUG:boto:path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:41.018 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj2 2024-04-09T21:52:41.018 DEBUG:boto:Method: GET 2024-04-09T21:52:41.018 DEBUG:boto:Path: /xryqqk-11/?prefix=obj2 2024-04-09T21:52:41.018 DEBUG:boto:Data: 2024-04-09T21:52:41.018 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.018 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.018 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.019 DEBUG:boto:Params: {} 2024-04-09T21:52:41.019 DEBUG:boto:Token: None 2024-04-09T21:52:41.019 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.019 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbbb23f907839eba-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.021 DEBUG:boto:b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-04-09T21:52:41.022 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.022 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.022 DEBUG:boto:path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.022 DEBUG:boto:auth_path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.022 DEBUG:boto:Method: GET 2024-04-09T21:52:41.023 DEBUG:boto:Path: /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.023 DEBUG:boto:Data: 2024-04-09T21:52:41.023 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.023 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.023 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.023 DEBUG:boto:Params: {} 2024-04-09T21:52:41.023 DEBUG:boto:Token: None 2024-04-09T21:52:41.023 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.023 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-09T21:52:41.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.025 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000607faf49bb9cf2f8-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.025 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.025 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.026 DEBUG:boto:path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.026 DEBUG:boto:auth_path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.026 DEBUG:boto:Method: GET 2024-04-09T21:52:41.026 DEBUG:boto:Path: /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.026 DEBUG:boto:Data: 2024-04-09T21:52:41.026 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.026 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.026 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.026 DEBUG:boto:Params: {} 2024-04-09T21:52:41.026 DEBUG:boto:Token: None 2024-04-09T21:52:41.027 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.027 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-09T21:52:41.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001368836f177820a4-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.029 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.029 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.030 DEBUG:boto:path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.030 DEBUG:boto:auth_path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.030 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.030 DEBUG:boto:Path: /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.030 DEBUG:boto:Data: 2024-04-09T21:52:41.030 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.030 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.030 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.030 DEBUG:boto:Params: {} 2024-04-09T21:52:41.030 DEBUG:boto:Token: None 2024-04-09T21:52:41.031 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.031 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4= 2024-04-09T21:52:41.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4=', 'Content-Length': '0'} 2024-04-09T21:52:41.032 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dee63fb8412e923-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.032 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.033 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.033 DEBUG:boto:path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.033 DEBUG:boto:auth_path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.033 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.033 DEBUG:boto:Path: /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.033 DEBUG:boto:Data: 2024-04-09T21:52:41.033 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.033 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.033 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.033 DEBUG:boto:Params: {} 2024-04-09T21:52:41.033 DEBUG:boto:Token: None 2024-04-09T21:52:41.034 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.034 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4= 2024-04-09T21:52:41.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4=', 'Content-Length': '0'} 2024-04-09T21:52:41.035 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005be374ad7475e780-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-04-09T21:52:41.035 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.036 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.036 DEBUG:boto:path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.036 DEBUG:boto:auth_path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.036 DEBUG:boto:Method: GET 2024-04-09T21:52:41.036 DEBUG:boto:Path: /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.036 DEBUG:boto:Data: 2024-04-09T21:52:41.036 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.036 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.036 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.036 DEBUG:boto:Params: {} 2024-04-09T21:52:41.036 DEBUG:boto:Token: None 2024-04-09T21:52:41.037 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.037 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-09T21:52:41.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fdc1f7159b9943b-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.039 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.039 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.039 DEBUG:boto:path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.039 DEBUG:boto:auth_path=/xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.039 DEBUG:boto:Method: GET 2024-04-09T21:52:41.039 DEBUG:boto:Path: /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.039 DEBUG:boto:Data: 2024-04-09T21:52:41.040 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.040 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.040 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.040 DEBUG:boto:Params: {} 2024-04-09T21:52:41.040 DEBUG:boto:Token: None 2024-04-09T21:52:41.040 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20?versionId=null 2024-04-09T21:52:41.040 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-09T21:52:41.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ee1829b3306694f-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.042 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.042 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.042 DEBUG:boto:path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.042 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.042 DEBUG:boto:Method: GET 2024-04-09T21:52:41.043 DEBUG:boto:Path: /xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.043 DEBUG:boto:Data: 2024-04-09T21:52:41.043 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.043 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.043 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.043 DEBUG:boto:Params: {} 2024-04-09T21:52:41.044 DEBUG:boto:Token: None 2024-04-09T21:52:41.044 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448c4837198251ce-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.090 DEBUG:boto:b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.091 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.091 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.091 DEBUG:boto:path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.091 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.091 DEBUG:boto:Method: GET 2024-04-09T21:52:41.091 DEBUG:boto:Path: /xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.091 DEBUG:boto:Data: 2024-04-09T21:52:41.092 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.092 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.092 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.092 DEBUG:boto:Params: {} 2024-04-09T21:52:41.092 DEBUG:boto:Token: None 2024-04-09T21:52:41.092 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.093 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000822447a9af2ac487-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.095 DEBUG:boto:b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.095 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.096 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.096 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.096 DEBUG:boto:Path: /xryqqk-11/obj20 2024-04-09T21:52:41.096 DEBUG:boto:Data: 2024-04-09T21:52:41.096 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.096 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.096 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.096 DEBUG:boto:Params: {} 2024-04-09T21:52:41.096 DEBUG:boto:Token: None 2024-04-09T21:52:41.096 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20 2024-04-09T21:52:41.097 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU= 2024-04-09T21:52:41.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU=', 'Content-Length': '0'} 2024-04-09T21:52:41.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000adebeb49ff9ca79-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.098 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.099 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.099 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.099 DEBUG:boto:Path: /xryqqk-11/obj20 2024-04-09T21:52:41.099 DEBUG:boto:Data: 2024-04-09T21:52:41.099 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.099 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.099 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.099 DEBUG:boto:Params: {} 2024-04-09T21:52:41.099 DEBUG:boto:Token: None 2024-04-09T21:52:41.099 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20 2024-04-09T21:52:41.100 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU= 2024-04-09T21:52:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU=', 'Content-Length': '0'} 2024-04-09T21:52:41.101 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a2750998d9e321f-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj20 2024-04-09T21:52:41.101 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.102 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.102 DEBUG:boto:Method: GET 2024-04-09T21:52:41.102 DEBUG:boto:Path: /xryqqk-11/obj20 2024-04-09T21:52:41.102 DEBUG:boto:Data: 2024-04-09T21:52:41.102 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.102 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.102 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.102 DEBUG:boto:Params: {} 2024-04-09T21:52:41.102 DEBUG:boto:Token: None 2024-04-09T21:52:41.103 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20 2024-04-09T21:52:41.103 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs= 2024-04-09T21:52:41.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs=', 'Content-Length': '0'} 2024-04-09T21:52:41.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052ba96dc6523ccb1-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.105 DEBUG:boto:path=/xryqqk-11/obj20 2024-04-09T21:52:41.105 DEBUG:boto:auth_path=/xryqqk-11/obj20 2024-04-09T21:52:41.105 DEBUG:boto:Method: GET 2024-04-09T21:52:41.105 DEBUG:boto:Path: /xryqqk-11/obj20 2024-04-09T21:52:41.106 DEBUG:boto:Data: 2024-04-09T21:52:41.106 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.106 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.106 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.106 DEBUG:boto:Params: {} 2024-04-09T21:52:41.106 DEBUG:boto:Token: None 2024-04-09T21:52:41.106 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj20 2024-04-09T21:52:41.106 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs= 2024-04-09T21:52:41.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs=', 'Content-Length': '0'} 2024-04-09T21:52:41.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4b920a1f93a21fc-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.108 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.109 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.109 DEBUG:boto:path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.109 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.109 DEBUG:boto:Method: GET 2024-04-09T21:52:41.109 DEBUG:boto:Path: /xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.109 DEBUG:boto:Data: 2024-04-09T21:52:41.109 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.109 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.109 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.109 DEBUG:boto:Params: {} 2024-04-09T21:52:41.109 DEBUG:boto:Token: None 2024-04-09T21:52:41.110 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.110 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005331f99a9f2d9217-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.112 DEBUG:boto:b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.113 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.113 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.113 DEBUG:boto:path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.113 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.113 DEBUG:boto:Method: GET 2024-04-09T21:52:41.113 DEBUG:boto:Path: /xryqqk-11/?prefix=obj20 2024-04-09T21:52:41.113 DEBUG:boto:Data: 2024-04-09T21:52:41.113 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.113 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.113 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.114 DEBUG:boto:Params: {} 2024-04-09T21:52:41.114 DEBUG:boto:Token: None 2024-04-09T21:52:41.114 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.114 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8bfb8036298bc63-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.117 DEBUG:boto:b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-04-09T21:52:41.118 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.118 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.118 DEBUG:boto:path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.118 DEBUG:boto:auth_path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.119 DEBUG:boto:Method: GET 2024-04-09T21:52:41.119 DEBUG:boto:Path: /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.119 DEBUG:boto:Data: 2024-04-09T21:52:41.119 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.119 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.119 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.120 DEBUG:boto:Params: {} 2024-04-09T21:52:41.120 DEBUG:boto:Token: None 2024-04-09T21:52:41.120 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.121 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-09T21:52:41.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-09T21:52:41.123 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb55a5087a38e1e5-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.124 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.124 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.124 DEBUG:boto:path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.124 DEBUG:boto:auth_path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.125 DEBUG:boto:Method: GET 2024-04-09T21:52:41.125 DEBUG:boto:Path: /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.125 DEBUG:boto:Data: 2024-04-09T21:52:41.125 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.125 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.125 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.126 DEBUG:boto:Params: {} 2024-04-09T21:52:41.126 DEBUG:boto:Token: None 2024-04-09T21:52:41.126 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.126 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-09T21:52:41.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-09T21:52:41.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000951f5b7a26e38c-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.129 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.129 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.130 DEBUG:boto:path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.130 DEBUG:boto:auth_path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.130 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.130 DEBUG:boto:Path: /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.130 DEBUG:boto:Data: 2024-04-09T21:52:41.131 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.131 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.131 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.131 DEBUG:boto:Params: {} 2024-04-09T21:52:41.131 DEBUG:boto:Token: None 2024-04-09T21:52:41.132 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.132 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4= 2024-04-09T21:52:41.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4=', 'Content-Length': '0'} 2024-04-09T21:52:41.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e198d3006a322de-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.134 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.135 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.135 DEBUG:boto:path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.135 DEBUG:boto:auth_path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.135 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.135 DEBUG:boto:Path: /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.136 DEBUG:boto:Data: 2024-04-09T21:52:41.136 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.136 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.136 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.136 DEBUG:boto:Params: {} 2024-04-09T21:52:41.137 DEBUG:boto:Token: None 2024-04-09T21:52:41.137 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.137 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4= 2024-04-09T21:52:41.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4=', 'Content-Length': '0'} 2024-04-09T21:52:41.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d2c10c4db51f71b-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-04-09T21:52:41.139 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.139 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.140 DEBUG:boto:path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.140 DEBUG:boto:auth_path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.140 DEBUG:boto:Method: GET 2024-04-09T21:52:41.140 DEBUG:boto:Path: /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.140 DEBUG:boto:Data: 2024-04-09T21:52:41.140 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.140 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.140 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.140 DEBUG:boto:Params: {} 2024-04-09T21:52:41.140 DEBUG:boto:Token: None 2024-04-09T21:52:41.141 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.141 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-09T21:52:41.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-09T21:52:41.142 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9bfd90b7d88ff41-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.143 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.143 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.143 DEBUG:boto:path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.143 DEBUG:boto:auth_path=/xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.143 DEBUG:boto:Method: GET 2024-04-09T21:52:41.143 DEBUG:boto:Path: /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.143 DEBUG:boto:Data: 2024-04-09T21:52:41.143 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.143 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.143 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.143 DEBUG:boto:Params: {} 2024-04-09T21:52:41.144 DEBUG:boto:Token: None 2024-04-09T21:52:41.144 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21?versionId=null 2024-04-09T21:52:41.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-09T21:52:41.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-09T21:52:41.145 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000600be81b5b389390-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.190 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.190 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.191 DEBUG:boto:path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.191 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.191 DEBUG:boto:Method: GET 2024-04-09T21:52:41.191 DEBUG:boto:Path: /xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.191 DEBUG:boto:Data: 2024-04-09T21:52:41.191 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.191 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.191 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.191 DEBUG:boto:Params: {} 2024-04-09T21:52:41.191 DEBUG:boto:Token: None 2024-04-09T21:52:41.192 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.192 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a81e29c809a816-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.194 DEBUG:boto:b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.195 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.195 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.195 DEBUG:boto:path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.195 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.195 DEBUG:boto:Method: GET 2024-04-09T21:52:41.195 DEBUG:boto:Path: /xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.195 DEBUG:boto:Data: 2024-04-09T21:52:41.196 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.196 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.196 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.196 DEBUG:boto:Params: {} 2024-04-09T21:52:41.196 DEBUG:boto:Token: None 2024-04-09T21:52:41.197 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.197 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e99265b170f82f-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.199 DEBUG:boto:b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.200 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.200 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.200 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.200 DEBUG:boto:Path: /xryqqk-11/obj21 2024-04-09T21:52:41.200 DEBUG:boto:Data: 2024-04-09T21:52:41.200 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.200 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.200 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.200 DEBUG:boto:Params: {} 2024-04-09T21:52:41.201 DEBUG:boto:Token: None 2024-04-09T21:52:41.201 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21 2024-04-09T21:52:41.201 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU= 2024-04-09T21:52:41.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU=', 'Content-Length': '0'} 2024-04-09T21:52:41.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a405699c2e9e1086-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.202 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.203 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.203 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.203 DEBUG:boto:Path: /xryqqk-11/obj21 2024-04-09T21:52:41.203 DEBUG:boto:Data: 2024-04-09T21:52:41.203 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.203 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.203 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.203 DEBUG:boto:Params: {} 2024-04-09T21:52:41.204 DEBUG:boto:Token: None 2024-04-09T21:52:41.204 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21 2024-04-09T21:52:41.204 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU= 2024-04-09T21:52:41.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU=', 'Content-Length': '0'} 2024-04-09T21:52:41.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c692bb8a7c6bdae-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj21 2024-04-09T21:52:41.206 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.206 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.206 DEBUG:boto:Method: GET 2024-04-09T21:52:41.206 DEBUG:boto:Path: /xryqqk-11/obj21 2024-04-09T21:52:41.206 DEBUG:boto:Data: 2024-04-09T21:52:41.206 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.206 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.206 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.206 DEBUG:boto:Params: {} 2024-04-09T21:52:41.207 DEBUG:boto:Token: None 2024-04-09T21:52:41.207 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21 2024-04-09T21:52:41.207 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc= 2024-04-09T21:52:41.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc=', 'Content-Length': '0'} 2024-04-09T21:52:41.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da1ff16465c5a4c7-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.250 DEBUG:boto:path=/xryqqk-11/obj21 2024-04-09T21:52:41.251 DEBUG:boto:auth_path=/xryqqk-11/obj21 2024-04-09T21:52:41.251 DEBUG:boto:Method: GET 2024-04-09T21:52:41.251 DEBUG:boto:Path: /xryqqk-11/obj21 2024-04-09T21:52:41.251 DEBUG:boto:Data: 2024-04-09T21:52:41.251 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.252 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.252 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.252 DEBUG:boto:Params: {} 2024-04-09T21:52:41.252 DEBUG:boto:Token: None 2024-04-09T21:52:41.252 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj21 2024-04-09T21:52:41.252 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc= 2024-04-09T21:52:41.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc=', 'Content-Length': '0'} 2024-04-09T21:52:41.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001123da2acd0164a9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.254 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.254 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.255 DEBUG:boto:path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.255 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.255 DEBUG:boto:Method: GET 2024-04-09T21:52:41.255 DEBUG:boto:Path: /xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.255 DEBUG:boto:Data: 2024-04-09T21:52:41.255 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.255 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.255 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.256 DEBUG:boto:Params: {} 2024-04-09T21:52:41.256 DEBUG:boto:Token: None 2024-04-09T21:52:41.256 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.256 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6f53dd9c1a3b88-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.259 DEBUG:boto:b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.259 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.260 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.260 DEBUG:boto:path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.260 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.260 DEBUG:boto:Method: GET 2024-04-09T21:52:41.260 DEBUG:boto:Path: /xryqqk-11/?prefix=obj21 2024-04-09T21:52:41.260 DEBUG:boto:Data: 2024-04-09T21:52:41.260 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.260 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.260 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.260 DEBUG:boto:Params: {} 2024-04-09T21:52:41.261 DEBUG:boto:Token: None 2024-04-09T21:52:41.261 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.261 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc55fa8ef02ed27-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.263 DEBUG:boto:b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-09T21:52:41.264 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.264 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.264 DEBUG:boto:path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.264 DEBUG:boto:auth_path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.264 DEBUG:boto:Method: GET 2024-04-09T21:52:41.264 DEBUG:boto:Path: /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.264 DEBUG:boto:Data: 2024-04-09T21:52:41.264 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.265 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.265 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.265 DEBUG:boto:Params: {} 2024-04-09T21:52:41.265 DEBUG:boto:Token: None 2024-04-09T21:52:41.265 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.265 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-09T21:52:41.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055c59dfe6afa45a4-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.267 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.267 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.267 DEBUG:boto:path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.267 DEBUG:boto:auth_path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.268 DEBUG:boto:Method: GET 2024-04-09T21:52:41.268 DEBUG:boto:Path: /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.268 DEBUG:boto:Data: 2024-04-09T21:52:41.268 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.268 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.268 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.268 DEBUG:boto:Params: {} 2024-04-09T21:52:41.268 DEBUG:boto:Token: None 2024-04-09T21:52:41.268 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.268 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-09T21:52:41.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.270 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df694db0f7357dd0-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.310 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.311 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.311 DEBUG:boto:path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.311 DEBUG:boto:auth_path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.311 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.311 DEBUG:boto:Path: /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.312 DEBUG:boto:Data: 2024-04-09T21:52:41.312 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.312 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.312 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.312 DEBUG:boto:Params: {} 2024-04-09T21:52:41.313 DEBUG:boto:Token: None 2024-04-09T21:52:41.313 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.313 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg= 2024-04-09T21:52:41.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg=', 'Content-Length': '0'} 2024-04-09T21:52:41.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cddc42c737e2bae-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.315 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.316 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.316 DEBUG:boto:path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.316 DEBUG:boto:auth_path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.316 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.316 DEBUG:boto:Path: /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.316 DEBUG:boto:Data: 2024-04-09T21:52:41.317 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.317 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.317 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.317 DEBUG:boto:Params: {} 2024-04-09T21:52:41.317 DEBUG:boto:Token: None 2024-04-09T21:52:41.317 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.318 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg= 2024-04-09T21:52:41.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg=', 'Content-Length': '0'} 2024-04-09T21:52:41.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004299e4af80833424-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-09T21:52:41.320 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.320 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.320 DEBUG:boto:path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.320 DEBUG:boto:auth_path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.320 DEBUG:boto:Method: GET 2024-04-09T21:52:41.320 DEBUG:boto:Path: /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.320 DEBUG:boto:Data: 2024-04-09T21:52:41.321 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.321 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.321 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.321 DEBUG:boto:Params: {} 2024-04-09T21:52:41.321 DEBUG:boto:Token: None 2024-04-09T21:52:41.321 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.321 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-09T21:52:41.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d87246f08b590186-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.323 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.324 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.324 DEBUG:boto:path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.324 DEBUG:boto:auth_path=/xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.324 DEBUG:boto:Method: GET 2024-04-09T21:52:41.324 DEBUG:boto:Path: /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.324 DEBUG:boto:Data: 2024-04-09T21:52:41.324 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.324 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.324 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.324 DEBUG:boto:Params: {} 2024-04-09T21:52:41.324 DEBUG:boto:Token: None 2024-04-09T21:52:41.325 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22?versionId=null 2024-04-09T21:52:41.325 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-09T21:52:41.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-09T21:52:41.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a88b740814d53c9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.326 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.327 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.327 DEBUG:boto:path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.327 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.327 DEBUG:boto:Method: GET 2024-04-09T21:52:41.327 DEBUG:boto:Path: /xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.327 DEBUG:boto:Data: 2024-04-09T21:52:41.327 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.327 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.327 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.327 DEBUG:boto:Params: {} 2024-04-09T21:52:41.327 DEBUG:boto:Token: None 2024-04-09T21:52:41.328 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.328 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000578e2c898fe49ca6-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.374 DEBUG:boto:b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.375 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.375 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.375 DEBUG:boto:path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.376 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.376 DEBUG:boto:Method: GET 2024-04-09T21:52:41.376 DEBUG:boto:Path: /xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.376 DEBUG:boto:Data: 2024-04-09T21:52:41.376 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.376 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.376 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.376 DEBUG:boto:Params: {} 2024-04-09T21:52:41.376 DEBUG:boto:Token: None 2024-04-09T21:52:41.377 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.377 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ace22c22b5f927fe-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.379 DEBUG:boto:b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.379 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.380 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.380 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.380 DEBUG:boto:Path: /xryqqk-11/obj22 2024-04-09T21:52:41.380 DEBUG:boto:Data: 2024-04-09T21:52:41.380 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.380 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.380 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.380 DEBUG:boto:Params: {} 2024-04-09T21:52:41.380 DEBUG:boto:Token: None 2024-04-09T21:52:41.380 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22 2024-04-09T21:52:41.381 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM= 2024-04-09T21:52:41.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM=', 'Content-Length': '0'} 2024-04-09T21:52:41.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009282b1583a89eee8-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.383 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.383 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.383 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.383 DEBUG:boto:Path: /xryqqk-11/obj22 2024-04-09T21:52:41.383 DEBUG:boto:Data: 2024-04-09T21:52:41.383 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.383 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.383 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.383 DEBUG:boto:Params: {} 2024-04-09T21:52:41.383 DEBUG:boto:Token: None 2024-04-09T21:52:41.384 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22 2024-04-09T21:52:41.384 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM= 2024-04-09T21:52:41.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM=', 'Content-Length': '0'} 2024-04-09T21:52:41.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000317f16e3d773fff9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-09T21:52:41.386 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.386 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.386 DEBUG:boto:Method: GET 2024-04-09T21:52:41.386 DEBUG:boto:Path: /xryqqk-11/obj22 2024-04-09T21:52:41.386 DEBUG:boto:Data: 2024-04-09T21:52:41.386 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.386 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.387 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.387 DEBUG:boto:Params: {} 2024-04-09T21:52:41.387 DEBUG:boto:Token: None 2024-04-09T21:52:41.387 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22 2024-04-09T21:52:41.387 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10= 2024-04-09T21:52:41.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10=', 'Content-Length': '0'} 2024-04-09T21:52:41.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000374cc55c45301e42-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.389 DEBUG:boto:path=/xryqqk-11/obj22 2024-04-09T21:52:41.389 DEBUG:boto:auth_path=/xryqqk-11/obj22 2024-04-09T21:52:41.389 DEBUG:boto:Method: GET 2024-04-09T21:52:41.389 DEBUG:boto:Path: /xryqqk-11/obj22 2024-04-09T21:52:41.389 DEBUG:boto:Data: 2024-04-09T21:52:41.390 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.390 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.390 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.390 DEBUG:boto:Params: {} 2024-04-09T21:52:41.390 DEBUG:boto:Token: None 2024-04-09T21:52:41.390 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj22 2024-04-09T21:52:41.390 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10= 2024-04-09T21:52:41.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10=', 'Content-Length': '0'} 2024-04-09T21:52:41.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7df85f6c82d8c51-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.392 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.392 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.392 DEBUG:boto:path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.392 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.392 DEBUG:boto:Method: GET 2024-04-09T21:52:41.392 DEBUG:boto:Path: /xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.393 DEBUG:boto:Data: 2024-04-09T21:52:41.393 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.393 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.393 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.393 DEBUG:boto:Params: {} 2024-04-09T21:52:41.393 DEBUG:boto:Token: None 2024-04-09T21:52:41.393 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.393 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006943ad6c0f50196f-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.396 DEBUG:boto:b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.396 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.396 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.396 DEBUG:boto:path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.396 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.396 DEBUG:boto:Method: GET 2024-04-09T21:52:41.397 DEBUG:boto:Path: /xryqqk-11/?prefix=obj22 2024-04-09T21:52:41.397 DEBUG:boto:Data: 2024-04-09T21:52:41.397 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.397 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.397 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.397 DEBUG:boto:Params: {} 2024-04-09T21:52:41.397 DEBUG:boto:Token: None 2024-04-09T21:52:41.397 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.397 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e5066b2e7e4061-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.400 DEBUG:boto:b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-09T21:52:41.400 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.400 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.400 DEBUG:boto:path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.400 DEBUG:boto:auth_path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.400 DEBUG:boto:Method: GET 2024-04-09T21:52:41.400 DEBUG:boto:Path: /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.401 DEBUG:boto:Data: 2024-04-09T21:52:41.401 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.401 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.401 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.401 DEBUG:boto:Params: {} 2024-04-09T21:52:41.401 DEBUG:boto:Token: None 2024-04-09T21:52:41.401 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.401 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-09T21:52:41.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-09T21:52:41.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041fe046e8fb31dba-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.446 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.449 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.449 DEBUG:boto:path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.449 DEBUG:boto:auth_path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.449 DEBUG:boto:Method: GET 2024-04-09T21:52:41.449 DEBUG:boto:Path: /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.449 DEBUG:boto:Data: 2024-04-09T21:52:41.450 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.450 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.450 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.450 DEBUG:boto:Params: {} 2024-04-09T21:52:41.450 DEBUG:boto:Token: None 2024-04-09T21:52:41.450 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.450 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-09T21:52:41.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-09T21:52:41.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f83a5f379fafb74-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.452 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.453 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.453 DEBUG:boto:path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.453 DEBUG:boto:auth_path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.453 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.453 DEBUG:boto:Path: /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.453 DEBUG:boto:Data: 2024-04-09T21:52:41.453 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.453 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.453 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.453 DEBUG:boto:Params: {} 2024-04-09T21:52:41.454 DEBUG:boto:Token: None 2024-04-09T21:52:41.454 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.454 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw= 2024-04-09T21:52:41.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw=', 'Content-Length': '0'} 2024-04-09T21:52:41.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003522b9a7210ad912-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.456 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.456 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.456 DEBUG:boto:path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.456 DEBUG:boto:auth_path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.456 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.456 DEBUG:boto:Path: /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.456 DEBUG:boto:Data: 2024-04-09T21:52:41.457 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.457 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.457 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.457 DEBUG:boto:Params: {} 2024-04-09T21:52:41.457 DEBUG:boto:Token: None 2024-04-09T21:52:41.457 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.457 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw= 2024-04-09T21:52:41.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw=', 'Content-Length': '0'} 2024-04-09T21:52:41.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001990df138607d202-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-09T21:52:41.459 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.459 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.459 DEBUG:boto:path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.460 DEBUG:boto:auth_path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.460 DEBUG:boto:Method: GET 2024-04-09T21:52:41.460 DEBUG:boto:Path: /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.460 DEBUG:boto:Data: 2024-04-09T21:52:41.460 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.460 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.460 DEBUG:boto:Params: {} 2024-04-09T21:52:41.460 DEBUG:boto:Token: None 2024-04-09T21:52:41.461 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.461 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-09T21:52:41.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-09T21:52:41.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bda72cda5e499662-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.463 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.463 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.463 DEBUG:boto:path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.463 DEBUG:boto:auth_path=/xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.463 DEBUG:boto:Method: GET 2024-04-09T21:52:41.463 DEBUG:boto:Path: /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.463 DEBUG:boto:Data: 2024-04-09T21:52:41.463 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.463 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.464 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.464 DEBUG:boto:Params: {} 2024-04-09T21:52:41.464 DEBUG:boto:Token: None 2024-04-09T21:52:41.464 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23?versionId=null 2024-04-09T21:52:41.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-09T21:52:41.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-09T21:52:41.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a86754cd8e6a8e4-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.515 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.515 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.515 DEBUG:boto:path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.515 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.515 DEBUG:boto:Method: GET 2024-04-09T21:52:41.515 DEBUG:boto:Path: /xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.515 DEBUG:boto:Data: 2024-04-09T21:52:41.515 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.515 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.515 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.516 DEBUG:boto:Params: {} 2024-04-09T21:52:41.516 DEBUG:boto:Token: None 2024-04-09T21:52:41.516 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.516 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8da02ac1e66283a-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.519 DEBUG:boto:b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.519 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.519 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.519 DEBUG:boto:path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.519 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.519 DEBUG:boto:Method: GET 2024-04-09T21:52:41.520 DEBUG:boto:Path: /xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.520 DEBUG:boto:Data: 2024-04-09T21:52:41.520 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.520 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.520 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.520 DEBUG:boto:Params: {} 2024-04-09T21:52:41.520 DEBUG:boto:Token: None 2024-04-09T21:52:41.521 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.521 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4754bfacfcc7fe9-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.523 DEBUG:boto:b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.524 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.524 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.524 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.524 DEBUG:boto:Path: /xryqqk-11/obj23 2024-04-09T21:52:41.524 DEBUG:boto:Data: 2024-04-09T21:52:41.524 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.524 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.524 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.524 DEBUG:boto:Params: {} 2024-04-09T21:52:41.525 DEBUG:boto:Token: None 2024-04-09T21:52:41.525 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23 2024-04-09T21:52:41.525 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY= 2024-04-09T21:52:41.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY=', 'Content-Length': '0'} 2024-04-09T21:52:41.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9ddf2c98d959b2-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.526 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.527 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.527 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.527 DEBUG:boto:Path: /xryqqk-11/obj23 2024-04-09T21:52:41.527 DEBUG:boto:Data: 2024-04-09T21:52:41.527 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.527 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.527 DEBUG:boto:Params: {} 2024-04-09T21:52:41.527 DEBUG:boto:Token: None 2024-04-09T21:52:41.527 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23 2024-04-09T21:52:41.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY= 2024-04-09T21:52:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY=', 'Content-Length': '0'} 2024-04-09T21:52:41.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c18b6f184c2abbf3-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-09T21:52:41.529 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.530 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.530 DEBUG:boto:Method: GET 2024-04-09T21:52:41.530 DEBUG:boto:Path: /xryqqk-11/obj23 2024-04-09T21:52:41.530 DEBUG:boto:Data: 2024-04-09T21:52:41.530 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.530 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.530 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.530 DEBUG:boto:Params: {} 2024-04-09T21:52:41.530 DEBUG:boto:Token: None 2024-04-09T21:52:41.530 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23 2024-04-09T21:52:41.531 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk= 2024-04-09T21:52:41.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk=', 'Content-Length': '0'} 2024-04-09T21:52:41.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d0895b5c919d53a-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.578 DEBUG:boto:path=/xryqqk-11/obj23 2024-04-09T21:52:41.578 DEBUG:boto:auth_path=/xryqqk-11/obj23 2024-04-09T21:52:41.579 DEBUG:boto:Method: GET 2024-04-09T21:52:41.579 DEBUG:boto:Path: /xryqqk-11/obj23 2024-04-09T21:52:41.579 DEBUG:boto:Data: 2024-04-09T21:52:41.579 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.579 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.579 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.579 DEBUG:boto:Params: {} 2024-04-09T21:52:41.579 DEBUG:boto:Token: None 2024-04-09T21:52:41.579 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj23 2024-04-09T21:52:41.580 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk= 2024-04-09T21:52:41.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk=', 'Content-Length': '0'} 2024-04-09T21:52:41.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc3b80ad9c0faeb0-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.582 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.582 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.582 DEBUG:boto:path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.582 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.582 DEBUG:boto:Method: GET 2024-04-09T21:52:41.582 DEBUG:boto:Path: /xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.582 DEBUG:boto:Data: 2024-04-09T21:52:41.582 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.583 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.583 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.583 DEBUG:boto:Params: {} 2024-04-09T21:52:41.583 DEBUG:boto:Token: None 2024-04-09T21:52:41.583 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.583 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a84fd33b002cf9e3-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.585 DEBUG:boto:b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.586 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.586 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.586 DEBUG:boto:path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.586 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.586 DEBUG:boto:Method: GET 2024-04-09T21:52:41.586 DEBUG:boto:Path: /xryqqk-11/?prefix=obj23 2024-04-09T21:52:41.586 DEBUG:boto:Data: 2024-04-09T21:52:41.587 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.587 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.587 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.587 DEBUG:boto:Params: {} 2024-04-09T21:52:41.587 DEBUG:boto:Token: None 2024-04-09T21:52:41.587 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.587 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004460b2cdbf7424a2-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.591 DEBUG:boto:b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-04-09T21:52:41.591 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.591 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.591 DEBUG:boto:path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.591 DEBUG:boto:auth_path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.591 DEBUG:boto:Method: GET 2024-04-09T21:52:41.592 DEBUG:boto:Path: /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.592 DEBUG:boto:Data: 2024-04-09T21:52:41.592 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.592 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.592 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.592 DEBUG:boto:Params: {} 2024-04-09T21:52:41.592 DEBUG:boto:Token: None 2024-04-09T21:52:41.592 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-09T21:52:41.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-09T21:52:41.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053089172e2e94002-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.594 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.595 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.595 DEBUG:boto:path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.595 DEBUG:boto:auth_path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.595 DEBUG:boto:Method: GET 2024-04-09T21:52:41.595 DEBUG:boto:Path: /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.595 DEBUG:boto:Data: 2024-04-09T21:52:41.595 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.595 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.595 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.595 DEBUG:boto:Params: {} 2024-04-09T21:52:41.595 DEBUG:boto:Token: None 2024-04-09T21:52:41.596 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.596 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-09T21:52:41.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-09T21:52:41.597 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3790ca7d27b6c70-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.597 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.598 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.598 DEBUG:boto:path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.598 DEBUG:boto:auth_path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.598 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.598 DEBUG:boto:Path: /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.598 DEBUG:boto:Data: 2024-04-09T21:52:41.598 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.598 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.598 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.598 DEBUG:boto:Params: {} 2024-04-09T21:52:41.599 DEBUG:boto:Token: None 2024-04-09T21:52:41.599 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.599 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0= 2024-04-09T21:52:41.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0=', 'Content-Length': '0'} 2024-04-09T21:52:41.600 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d34c6e2f3080639-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.601 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.601 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.601 DEBUG:boto:path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.601 DEBUG:boto:auth_path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.601 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.601 DEBUG:boto:Path: /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.602 DEBUG:boto:Data: 2024-04-09T21:52:41.602 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.602 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.602 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.602 DEBUG:boto:Params: {} 2024-04-09T21:52:41.602 DEBUG:boto:Token: None 2024-04-09T21:52:41.602 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.602 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0= 2024-04-09T21:52:41.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0=', 'Content-Length': '0'} 2024-04-09T21:52:41.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000208367efa42c3e60-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-04-09T21:52:41.604 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.604 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.605 DEBUG:boto:path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.605 DEBUG:boto:auth_path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.605 DEBUG:boto:Method: GET 2024-04-09T21:52:41.605 DEBUG:boto:Path: /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.605 DEBUG:boto:Data: 2024-04-09T21:52:41.605 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.605 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.605 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.605 DEBUG:boto:Params: {} 2024-04-09T21:52:41.605 DEBUG:boto:Token: None 2024-04-09T21:52:41.605 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.606 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-09T21:52:41.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-09T21:52:41.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc99a8b5e5c35960-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.608 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.608 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.608 DEBUG:boto:path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.608 DEBUG:boto:auth_path=/xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.608 DEBUG:boto:Method: GET 2024-04-09T21:52:41.608 DEBUG:boto:Path: /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.608 DEBUG:boto:Data: 2024-04-09T21:52:41.608 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.608 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.609 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.609 DEBUG:boto:Params: {} 2024-04-09T21:52:41.609 DEBUG:boto:Token: None 2024-04-09T21:52:41.609 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24?versionId=null 2024-04-09T21:52:41.609 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-09T21:52:41.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-09T21:52:41.611 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b0103803316d687-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.612 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.612 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.612 DEBUG:boto:path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.612 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.612 DEBUG:boto:Method: GET 2024-04-09T21:52:41.612 DEBUG:boto:Path: /xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.612 DEBUG:boto:Data: 2024-04-09T21:52:41.612 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.612 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.613 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.613 DEBUG:boto:Params: {} 2024-04-09T21:52:41.613 DEBUG:boto:Token: None 2024-04-09T21:52:41.613 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.613 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089785763d18385cf-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.616 DEBUG:boto:b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.616 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.616 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.616 DEBUG:boto:path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.616 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.616 DEBUG:boto:Method: GET 2024-04-09T21:52:41.616 DEBUG:boto:Path: /xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.617 DEBUG:boto:Data: 2024-04-09T21:52:41.617 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.617 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.617 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.617 DEBUG:boto:Params: {} 2024-04-09T21:52:41.617 DEBUG:boto:Token: None 2024-04-09T21:52:41.617 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.617 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.619 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c85f497f2e2a99ba-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.666 DEBUG:boto:b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.667 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.667 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.667 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.667 DEBUG:boto:Path: /xryqqk-11/obj24 2024-04-09T21:52:41.667 DEBUG:boto:Data: 2024-04-09T21:52:41.667 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.667 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.668 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.668 DEBUG:boto:Params: {} 2024-04-09T21:52:41.668 DEBUG:boto:Token: None 2024-04-09T21:52:41.668 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24 2024-04-09T21:52:41.668 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk= 2024-04-09T21:52:41.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk=', 'Content-Length': '0'} 2024-04-09T21:52:41.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095dce07e58bc0554-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.670 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.670 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.670 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.670 DEBUG:boto:Path: /xryqqk-11/obj24 2024-04-09T21:52:41.671 DEBUG:boto:Data: 2024-04-09T21:52:41.671 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.671 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.671 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.671 DEBUG:boto:Params: {} 2024-04-09T21:52:41.671 DEBUG:boto:Token: None 2024-04-09T21:52:41.671 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24 2024-04-09T21:52:41.671 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk= 2024-04-09T21:52:41.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk=', 'Content-Length': '0'} 2024-04-09T21:52:41.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c777546bb8b4c6a9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj24 2024-04-09T21:52:41.673 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.673 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.674 DEBUG:boto:Method: GET 2024-04-09T21:52:41.674 DEBUG:boto:Path: /xryqqk-11/obj24 2024-04-09T21:52:41.674 DEBUG:boto:Data: 2024-04-09T21:52:41.674 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.674 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.674 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.674 DEBUG:boto:Params: {} 2024-04-09T21:52:41.674 DEBUG:boto:Token: None 2024-04-09T21:52:41.674 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24 2024-04-09T21:52:41.675 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE= 2024-04-09T21:52:41.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE=', 'Content-Length': '0'} 2024-04-09T21:52:41.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e034a2623e52ce70-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.677 DEBUG:boto:path=/xryqqk-11/obj24 2024-04-09T21:52:41.677 DEBUG:boto:auth_path=/xryqqk-11/obj24 2024-04-09T21:52:41.677 DEBUG:boto:Method: GET 2024-04-09T21:52:41.677 DEBUG:boto:Path: /xryqqk-11/obj24 2024-04-09T21:52:41.677 DEBUG:boto:Data: 2024-04-09T21:52:41.677 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.677 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.677 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.677 DEBUG:boto:Params: {} 2024-04-09T21:52:41.677 DEBUG:boto:Token: None 2024-04-09T21:52:41.678 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj24 2024-04-09T21:52:41.678 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE= 2024-04-09T21:52:41.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE=', 'Content-Length': '0'} 2024-04-09T21:52:41.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077cd6d4bc0513f19-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.680 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.680 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.680 DEBUG:boto:path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.681 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.681 DEBUG:boto:Method: GET 2024-04-09T21:52:41.681 DEBUG:boto:Path: /xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.681 DEBUG:boto:Data: 2024-04-09T21:52:41.681 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.681 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.681 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.681 DEBUG:boto:Params: {} 2024-04-09T21:52:41.681 DEBUG:boto:Token: None 2024-04-09T21:52:41.681 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.682 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ee2a131461444e-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.685 DEBUG:boto:b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.685 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.685 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.685 DEBUG:boto:path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.685 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.685 DEBUG:boto:Method: GET 2024-04-09T21:52:41.686 DEBUG:boto:Path: /xryqqk-11/?prefix=obj24 2024-04-09T21:52:41.686 DEBUG:boto:Data: 2024-04-09T21:52:41.686 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.686 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.686 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.686 DEBUG:boto:Params: {} 2024-04-09T21:52:41.686 DEBUG:boto:Token: None 2024-04-09T21:52:41.686 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.686 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dd2943ae0b96a4b-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.689 DEBUG:boto:b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-04-09T21:52:41.690 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.690 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.690 DEBUG:boto:path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.690 DEBUG:boto:auth_path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.690 DEBUG:boto:Method: GET 2024-04-09T21:52:41.690 DEBUG:boto:Path: /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.690 DEBUG:boto:Data: 2024-04-09T21:52:41.690 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.690 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.690 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.690 DEBUG:boto:Params: {} 2024-04-09T21:52:41.691 DEBUG:boto:Token: None 2024-04-09T21:52:41.691 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.691 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-09T21:52:41.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-09T21:52:41.693 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000810d92113fad3bd4-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.693 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.693 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.693 DEBUG:boto:path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.693 DEBUG:boto:auth_path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.693 DEBUG:boto:Method: GET 2024-04-09T21:52:41.694 DEBUG:boto:Path: /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.694 DEBUG:boto:Data: 2024-04-09T21:52:41.694 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.694 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.694 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.694 DEBUG:boto:Params: {} 2024-04-09T21:52:41.694 DEBUG:boto:Token: None 2024-04-09T21:52:41.694 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.694 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-09T21:52:41.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-09T21:52:41.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d75f2ba5f500677-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.696 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.697 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.697 DEBUG:boto:path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.697 DEBUG:boto:auth_path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.697 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.697 DEBUG:boto:Path: /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.697 DEBUG:boto:Data: 2024-04-09T21:52:41.697 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.697 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.697 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.697 DEBUG:boto:Params: {} 2024-04-09T21:52:41.697 DEBUG:boto:Token: None 2024-04-09T21:52:41.698 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.698 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q= 2024-04-09T21:52:41.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q=', 'Content-Length': '0'} 2024-04-09T21:52:41.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfdd7a836bba1a37-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.699 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.699 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.700 DEBUG:boto:path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.700 DEBUG:boto:auth_path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.700 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.700 DEBUG:boto:Path: /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.700 DEBUG:boto:Data: 2024-04-09T21:52:41.700 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.700 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.700 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.700 DEBUG:boto:Params: {} 2024-04-09T21:52:41.700 DEBUG:boto:Token: None 2024-04-09T21:52:41.700 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.701 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q= 2024-04-09T21:52:41.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q=', 'Content-Length': '0'} 2024-04-09T21:52:41.702 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054399c7653b93abe-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-04-09T21:52:41.703 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.703 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.703 DEBUG:boto:path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.703 DEBUG:boto:auth_path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.703 DEBUG:boto:Method: GET 2024-04-09T21:52:41.703 DEBUG:boto:Path: /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.703 DEBUG:boto:Data: 2024-04-09T21:52:41.703 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.703 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.703 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.703 DEBUG:boto:Params: {} 2024-04-09T21:52:41.704 DEBUG:boto:Token: None 2024-04-09T21:52:41.704 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.704 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-09T21:52:41.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-09T21:52:41.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f74aea9fd1bb820b-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.706 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.706 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.706 DEBUG:boto:path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.706 DEBUG:boto:auth_path=/xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.706 DEBUG:boto:Method: GET 2024-04-09T21:52:41.706 DEBUG:boto:Path: /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.706 DEBUG:boto:Data: 2024-04-09T21:52:41.707 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.707 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.707 DEBUG:boto:Params: {} 2024-04-09T21:52:41.707 DEBUG:boto:Token: None 2024-04-09T21:52:41.707 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25?versionId=null 2024-04-09T21:52:41.707 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-09T21:52:41.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-09T21:52:41.709 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004741981ca9cddd6c-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.709 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.709 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.709 DEBUG:boto:path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.709 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.709 DEBUG:boto:Method: GET 2024-04-09T21:52:41.709 DEBUG:boto:Path: /xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.709 DEBUG:boto:Data: 2024-04-09T21:52:41.710 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.710 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.710 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.710 DEBUG:boto:Params: {} 2024-04-09T21:52:41.710 DEBUG:boto:Token: None 2024-04-09T21:52:41.710 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.710 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d96706f89328d8-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.713 DEBUG:boto:b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.713 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.713 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.713 DEBUG:boto:path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.713 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.713 DEBUG:boto:Method: GET 2024-04-09T21:52:41.713 DEBUG:boto:Path: /xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.714 DEBUG:boto:Data: 2024-04-09T21:52:41.714 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.714 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.714 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.714 DEBUG:boto:Params: {} 2024-04-09T21:52:41.714 DEBUG:boto:Token: None 2024-04-09T21:52:41.714 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.714 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e019626fc4bbcb3-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.716 DEBUG:boto:b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.717 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.717 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.717 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.717 DEBUG:boto:Path: /xryqqk-11/obj25 2024-04-09T21:52:41.717 DEBUG:boto:Data: 2024-04-09T21:52:41.717 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.717 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.717 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.717 DEBUG:boto:Params: {} 2024-04-09T21:52:41.718 DEBUG:boto:Token: None 2024-04-09T21:52:41.718 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25 2024-04-09T21:52:41.718 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE= 2024-04-09T21:52:41.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE=', 'Content-Length': '0'} 2024-04-09T21:52:41.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000059d47a49b202a75-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.720 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.720 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.720 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.720 DEBUG:boto:Path: /xryqqk-11/obj25 2024-04-09T21:52:41.720 DEBUG:boto:Data: 2024-04-09T21:52:41.720 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.720 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.720 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.720 DEBUG:boto:Params: {} 2024-04-09T21:52:41.720 DEBUG:boto:Token: None 2024-04-09T21:52:41.721 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25 2024-04-09T21:52:41.721 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE= 2024-04-09T21:52:41.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE=', 'Content-Length': '0'} 2024-04-09T21:52:41.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f234521ac9275761-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj25 2024-04-09T21:52:41.723 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.723 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.723 DEBUG:boto:Method: GET 2024-04-09T21:52:41.723 DEBUG:boto:Path: /xryqqk-11/obj25 2024-04-09T21:52:41.723 DEBUG:boto:Data: 2024-04-09T21:52:41.723 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.723 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.723 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.723 DEBUG:boto:Params: {} 2024-04-09T21:52:41.723 DEBUG:boto:Token: None 2024-04-09T21:52:41.724 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25 2024-04-09T21:52:41.724 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g= 2024-04-09T21:52:41.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g=', 'Content-Length': '0'} 2024-04-09T21:52:41.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef6fd78771ba865-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.725 DEBUG:boto:path=/xryqqk-11/obj25 2024-04-09T21:52:41.725 DEBUG:boto:auth_path=/xryqqk-11/obj25 2024-04-09T21:52:41.726 DEBUG:boto:Method: GET 2024-04-09T21:52:41.726 DEBUG:boto:Path: /xryqqk-11/obj25 2024-04-09T21:52:41.726 DEBUG:boto:Data: 2024-04-09T21:52:41.726 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.726 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.726 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.726 DEBUG:boto:Params: {} 2024-04-09T21:52:41.726 DEBUG:boto:Token: None 2024-04-09T21:52:41.726 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj25 2024-04-09T21:52:41.727 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g= 2024-04-09T21:52:41.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g=', 'Content-Length': '0'} 2024-04-09T21:52:41.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2351ee9922f1f91-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.729 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.729 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.729 DEBUG:boto:path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.729 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.729 DEBUG:boto:Method: GET 2024-04-09T21:52:41.729 DEBUG:boto:Path: /xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.729 DEBUG:boto:Data: 2024-04-09T21:52:41.729 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.729 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.729 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.730 DEBUG:boto:Params: {} 2024-04-09T21:52:41.730 DEBUG:boto:Token: None 2024-04-09T21:52:41.730 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.730 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048de3531c5750625-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.732 DEBUG:boto:b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.732 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.732 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.733 DEBUG:boto:path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.733 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.733 DEBUG:boto:Method: GET 2024-04-09T21:52:41.733 DEBUG:boto:Path: /xryqqk-11/?prefix=obj25 2024-04-09T21:52:41.733 DEBUG:boto:Data: 2024-04-09T21:52:41.733 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.733 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.733 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.733 DEBUG:boto:Params: {} 2024-04-09T21:52:41.733 DEBUG:boto:Token: None 2024-04-09T21:52:41.733 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.734 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931d1ca0ec130bfb-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.736 DEBUG:boto:b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-04-09T21:52:41.736 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.736 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.736 DEBUG:boto:path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.736 DEBUG:boto:auth_path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.737 DEBUG:boto:Method: GET 2024-04-09T21:52:41.737 DEBUG:boto:Path: /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.737 DEBUG:boto:Data: 2024-04-09T21:52:41.737 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.737 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.737 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.737 DEBUG:boto:Params: {} 2024-04-09T21:52:41.737 DEBUG:boto:Token: None 2024-04-09T21:52:41.737 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.737 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-09T21:52:41.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-09T21:52:41.739 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab94dae7c50e7a9a-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.739 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.739 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.739 DEBUG:boto:path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.739 DEBUG:boto:auth_path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.740 DEBUG:boto:Method: GET 2024-04-09T21:52:41.740 DEBUG:boto:Path: /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.740 DEBUG:boto:Data: 2024-04-09T21:52:41.740 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.740 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.740 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.740 DEBUG:boto:Params: {} 2024-04-09T21:52:41.740 DEBUG:boto:Token: None 2024-04-09T21:52:41.740 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.740 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-09T21:52:41.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-09T21:52:41.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044547071f5f423a3-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.742 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.742 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.742 DEBUG:boto:path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.742 DEBUG:boto:auth_path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.743 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.743 DEBUG:boto:Path: /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.743 DEBUG:boto:Data: 2024-04-09T21:52:41.743 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.743 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.743 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.743 DEBUG:boto:Params: {} 2024-04-09T21:52:41.743 DEBUG:boto:Token: None 2024-04-09T21:52:41.743 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.743 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo= 2024-04-09T21:52:41.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo=', 'Content-Length': '0'} 2024-04-09T21:52:41.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f5d98c4563b07b6-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.745 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.745 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.745 DEBUG:boto:path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.745 DEBUG:boto:auth_path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.745 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.746 DEBUG:boto:Path: /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.746 DEBUG:boto:Data: 2024-04-09T21:52:41.746 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.746 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.746 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.746 DEBUG:boto:Params: {} 2024-04-09T21:52:41.746 DEBUG:boto:Token: None 2024-04-09T21:52:41.746 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.746 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo= 2024-04-09T21:52:41.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo=', 'Content-Length': '0'} 2024-04-09T21:52:41.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216eb56550d3c3c8-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-04-09T21:52:41.748 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.748 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.748 DEBUG:boto:path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.749 DEBUG:boto:auth_path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.749 DEBUG:boto:Method: GET 2024-04-09T21:52:41.749 DEBUG:boto:Path: /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.749 DEBUG:boto:Data: 2024-04-09T21:52:41.749 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.749 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.749 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.749 DEBUG:boto:Params: {} 2024-04-09T21:52:41.749 DEBUG:boto:Token: None 2024-04-09T21:52:41.749 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.750 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-09T21:52:41.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-09T21:52:41.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f6d270570868091-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.795 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.795 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.795 DEBUG:boto:path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.795 DEBUG:boto:auth_path=/xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.795 DEBUG:boto:Method: GET 2024-04-09T21:52:41.795 DEBUG:boto:Path: /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.795 DEBUG:boto:Data: 2024-04-09T21:52:41.796 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.796 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.796 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.796 DEBUG:boto:Params: {} 2024-04-09T21:52:41.796 DEBUG:boto:Token: None 2024-04-09T21:52:41.796 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26?versionId=null 2024-04-09T21:52:41.796 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-09T21:52:41.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-09T21:52:41.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f450fb4a101f41c-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.798 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.798 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.799 DEBUG:boto:path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.799 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.799 DEBUG:boto:Method: GET 2024-04-09T21:52:41.799 DEBUG:boto:Path: /xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.799 DEBUG:boto:Data: 2024-04-09T21:52:41.799 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.799 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.799 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.799 DEBUG:boto:Params: {} 2024-04-09T21:52:41.799 DEBUG:boto:Token: None 2024-04-09T21:52:41.799 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.800 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003828734feeb40206-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.802 DEBUG:boto:b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.802 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.803 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.803 DEBUG:boto:path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.803 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.803 DEBUG:boto:Method: GET 2024-04-09T21:52:41.803 DEBUG:boto:Path: /xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.803 DEBUG:boto:Data: 2024-04-09T21:52:41.803 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.803 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.803 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.803 DEBUG:boto:Params: {} 2024-04-09T21:52:41.803 DEBUG:boto:Token: None 2024-04-09T21:52:41.804 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b58be1a70ef32eb-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.806 DEBUG:boto:b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.806 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.806 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.807 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.807 DEBUG:boto:Path: /xryqqk-11/obj26 2024-04-09T21:52:41.808 DEBUG:boto:Data: 2024-04-09T21:52:41.808 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.808 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.808 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.808 DEBUG:boto:Params: {} 2024-04-09T21:52:41.808 DEBUG:boto:Token: None 2024-04-09T21:52:41.808 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26 2024-04-09T21:52:41.808 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg= 2024-04-09T21:52:41.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg=', 'Content-Length': '0'} 2024-04-09T21:52:41.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b2201f21ed4b99d-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.810 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.810 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.810 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.810 DEBUG:boto:Path: /xryqqk-11/obj26 2024-04-09T21:52:41.810 DEBUG:boto:Data: 2024-04-09T21:52:41.811 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.811 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.811 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.811 DEBUG:boto:Params: {} 2024-04-09T21:52:41.811 DEBUG:boto:Token: None 2024-04-09T21:52:41.811 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26 2024-04-09T21:52:41.811 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg= 2024-04-09T21:52:41.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg=', 'Content-Length': '0'} 2024-04-09T21:52:41.813 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf9271a9f0823e8b-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj26 2024-04-09T21:52:41.813 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.813 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.813 DEBUG:boto:Method: GET 2024-04-09T21:52:41.813 DEBUG:boto:Path: /xryqqk-11/obj26 2024-04-09T21:52:41.813 DEBUG:boto:Data: 2024-04-09T21:52:41.813 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.813 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.813 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.814 DEBUG:boto:Params: {} 2024-04-09T21:52:41.814 DEBUG:boto:Token: None 2024-04-09T21:52:41.814 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26 2024-04-09T21:52:41.814 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8= 2024-04-09T21:52:41.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8=', 'Content-Length': '0'} 2024-04-09T21:52:41.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e124ef327b177386-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.816 DEBUG:boto:path=/xryqqk-11/obj26 2024-04-09T21:52:41.816 DEBUG:boto:auth_path=/xryqqk-11/obj26 2024-04-09T21:52:41.816 DEBUG:boto:Method: GET 2024-04-09T21:52:41.816 DEBUG:boto:Path: /xryqqk-11/obj26 2024-04-09T21:52:41.816 DEBUG:boto:Data: 2024-04-09T21:52:41.816 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.817 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.817 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.817 DEBUG:boto:Params: {} 2024-04-09T21:52:41.817 DEBUG:boto:Token: None 2024-04-09T21:52:41.817 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj26 2024-04-09T21:52:41.817 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8= 2024-04-09T21:52:41.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8=', 'Content-Length': '0'} 2024-04-09T21:52:41.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007385bc8ed42ac28-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.819 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.819 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.819 DEBUG:boto:path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.819 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.819 DEBUG:boto:Method: GET 2024-04-09T21:52:41.819 DEBUG:boto:Path: /xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.819 DEBUG:boto:Data: 2024-04-09T21:52:41.819 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.820 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.820 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.820 DEBUG:boto:Params: {} 2024-04-09T21:52:41.820 DEBUG:boto:Token: None 2024-04-09T21:52:41.820 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.820 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a59097dcdb51c7-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.822 DEBUG:boto:b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.823 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.823 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.823 DEBUG:boto:path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.823 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.823 DEBUG:boto:Method: GET 2024-04-09T21:52:41.823 DEBUG:boto:Path: /xryqqk-11/?prefix=obj26 2024-04-09T21:52:41.823 DEBUG:boto:Data: 2024-04-09T21:52:41.823 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.824 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.824 DEBUG:boto:Params: {} 2024-04-09T21:52:41.824 DEBUG:boto:Token: None 2024-04-09T21:52:41.824 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cf26ac06ab5c9ac-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.866 DEBUG:boto:b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-04-09T21:52:41.867 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.867 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.867 DEBUG:boto:path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.867 DEBUG:boto:auth_path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.867 DEBUG:boto:Method: GET 2024-04-09T21:52:41.867 DEBUG:boto:Path: /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.868 DEBUG:boto:Data: 2024-04-09T21:52:41.868 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.868 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.868 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.868 DEBUG:boto:Params: {} 2024-04-09T21:52:41.868 DEBUG:boto:Token: None 2024-04-09T21:52:41.868 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.868 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-09T21:52:41.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-09T21:52:41.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049ae5aa904cc0206-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.870 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.871 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.871 DEBUG:boto:path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.871 DEBUG:boto:auth_path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.871 DEBUG:boto:Method: GET 2024-04-09T21:52:41.871 DEBUG:boto:Path: /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.871 DEBUG:boto:Data: 2024-04-09T21:52:41.871 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.871 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.871 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.871 DEBUG:boto:Params: {} 2024-04-09T21:52:41.871 DEBUG:boto:Token: None 2024-04-09T21:52:41.872 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.872 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-09T21:52:41.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-09T21:52:41.873 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015618b491878f5ae-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.873 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.874 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.874 DEBUG:boto:path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.874 DEBUG:boto:auth_path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.874 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.874 DEBUG:boto:Path: /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.874 DEBUG:boto:Data: 2024-04-09T21:52:41.874 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.874 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.874 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.874 DEBUG:boto:Params: {} 2024-04-09T21:52:41.875 DEBUG:boto:Token: None 2024-04-09T21:52:41.875 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.875 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM= 2024-04-09T21:52:41.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM=', 'Content-Length': '0'} 2024-04-09T21:52:41.876 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d608077dfa4f7a8-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.877 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.877 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.877 DEBUG:boto:path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.877 DEBUG:boto:auth_path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.877 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.877 DEBUG:boto:Path: /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.877 DEBUG:boto:Data: 2024-04-09T21:52:41.877 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.877 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.877 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.877 DEBUG:boto:Params: {} 2024-04-09T21:52:41.878 DEBUG:boto:Token: None 2024-04-09T21:52:41.878 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.878 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM= 2024-04-09T21:52:41.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM=', 'Content-Length': '0'} 2024-04-09T21:52:41.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fba2fdd9250e2c7-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-04-09T21:52:41.880 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.880 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.880 DEBUG:boto:path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.880 DEBUG:boto:auth_path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.880 DEBUG:boto:Method: GET 2024-04-09T21:52:41.880 DEBUG:boto:Path: /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.880 DEBUG:boto:Data: 2024-04-09T21:52:41.880 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.880 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.880 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.880 DEBUG:boto:Params: {} 2024-04-09T21:52:41.881 DEBUG:boto:Token: None 2024-04-09T21:52:41.881 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.881 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-09T21:52:41.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-09T21:52:41.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046d8ffaf0591c4da-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.883 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.883 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.883 DEBUG:boto:path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.883 DEBUG:boto:auth_path=/xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.883 DEBUG:boto:Method: GET 2024-04-09T21:52:41.883 DEBUG:boto:Path: /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.883 DEBUG:boto:Data: 2024-04-09T21:52:41.883 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.883 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.883 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.883 DEBUG:boto:Params: {} 2024-04-09T21:52:41.884 DEBUG:boto:Token: None 2024-04-09T21:52:41.884 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27?versionId=null 2024-04-09T21:52:41.884 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-09T21:52:41.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-09T21:52:41.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003161fab8699f98c8-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.886 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.886 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.886 DEBUG:boto:path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.886 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.886 DEBUG:boto:Method: GET 2024-04-09T21:52:41.886 DEBUG:boto:Path: /xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.886 DEBUG:boto:Data: 2024-04-09T21:52:41.886 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.886 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.886 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.887 DEBUG:boto:Params: {} 2024-04-09T21:52:41.887 DEBUG:boto:Token: None 2024-04-09T21:52:41.887 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.887 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008693e3e2145edae9-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.930 DEBUG:boto:b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.931 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.931 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.931 DEBUG:boto:path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.931 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.931 DEBUG:boto:Method: GET 2024-04-09T21:52:41.931 DEBUG:boto:Path: /xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.931 DEBUG:boto:Data: 2024-04-09T21:52:41.932 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.932 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.932 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.932 DEBUG:boto:Params: {} 2024-04-09T21:52:41.932 DEBUG:boto:Token: None 2024-04-09T21:52:41.932 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.932 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4bbc3f77391a654-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.935 DEBUG:boto:b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.935 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.935 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.935 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.935 DEBUG:boto:Path: /xryqqk-11/obj27 2024-04-09T21:52:41.935 DEBUG:boto:Data: 2024-04-09T21:52:41.935 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.936 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.936 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.936 DEBUG:boto:Params: {} 2024-04-09T21:52:41.936 DEBUG:boto:Token: None 2024-04-09T21:52:41.936 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27 2024-04-09T21:52:41.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y= 2024-04-09T21:52:41.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y=', 'Content-Length': '0'} 2024-04-09T21:52:41.938 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f7ec84b7b24ea47-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.938 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.938 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.938 DEBUG:boto:Method: HEAD 2024-04-09T21:52:41.938 DEBUG:boto:Path: /xryqqk-11/obj27 2024-04-09T21:52:41.938 DEBUG:boto:Data: 2024-04-09T21:52:41.938 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.938 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.939 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.939 DEBUG:boto:Params: {} 2024-04-09T21:52:41.939 DEBUG:boto:Token: None 2024-04-09T21:52:41.939 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27 2024-04-09T21:52:41.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y= 2024-04-09T21:52:41.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y=', 'Content-Length': '0'} 2024-04-09T21:52:41.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063af9af0ad0d8983-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj27 2024-04-09T21:52:41.941 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.941 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.941 DEBUG:boto:Method: GET 2024-04-09T21:52:41.941 DEBUG:boto:Path: /xryqqk-11/obj27 2024-04-09T21:52:41.941 DEBUG:boto:Data: 2024-04-09T21:52:41.941 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.941 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.941 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.941 DEBUG:boto:Params: {} 2024-04-09T21:52:41.941 DEBUG:boto:Token: None 2024-04-09T21:52:41.942 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27 2024-04-09T21:52:41.942 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg= 2024-04-09T21:52:41.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg=', 'Content-Length': '0'} 2024-04-09T21:52:41.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f61f4375249b271-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.944 DEBUG:boto:path=/xryqqk-11/obj27 2024-04-09T21:52:41.944 DEBUG:boto:auth_path=/xryqqk-11/obj27 2024-04-09T21:52:41.944 DEBUG:boto:Method: GET 2024-04-09T21:52:41.944 DEBUG:boto:Path: /xryqqk-11/obj27 2024-04-09T21:52:41.944 DEBUG:boto:Data: 2024-04-09T21:52:41.944 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.944 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.944 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.944 DEBUG:boto:Params: {} 2024-04-09T21:52:41.944 DEBUG:boto:Token: None 2024-04-09T21:52:41.945 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj27 2024-04-09T21:52:41.945 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg= 2024-04-09T21:52:41.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg=', 'Content-Length': '0'} 2024-04-09T21:52:41.946 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efd5bac6b25f5905-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.946 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.947 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.947 DEBUG:boto:path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.947 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.947 DEBUG:boto:Method: GET 2024-04-09T21:52:41.947 DEBUG:boto:Path: /xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.947 DEBUG:boto:Data: 2024-04-09T21:52:41.947 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.947 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.947 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.947 DEBUG:boto:Params: {} 2024-04-09T21:52:41.947 DEBUG:boto:Token: None 2024-04-09T21:52:41.948 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.948 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f6fc2da47441d5-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.950 DEBUG:boto:b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.951 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:41.951 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:41.951 DEBUG:boto:path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.951 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.951 DEBUG:boto:Method: GET 2024-04-09T21:52:41.951 DEBUG:boto:Path: /xryqqk-11/?prefix=obj27 2024-04-09T21:52:41.951 DEBUG:boto:Data: 2024-04-09T21:52:41.951 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.951 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.951 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.952 DEBUG:boto:Params: {} 2024-04-09T21:52:41.952 DEBUG:boto:Token: None 2024-04-09T21:52:41.952 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/ 2024-04-09T21:52:41.952 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-09T21:52:41.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-09T21:52:41.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028810939bd05c3f6-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.955 DEBUG:boto:b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:41.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-04-09T21:52:41.955 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:41.955 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:41.955 DEBUG:boto:path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.955 DEBUG:boto:auth_path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.956 DEBUG:boto:Method: GET 2024-04-09T21:52:41.956 DEBUG:boto:Path: /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.956 DEBUG:boto:Data: 2024-04-09T21:52:41.956 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.956 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:41.956 DEBUG:boto:Port: 8001 2024-04-09T21:52:41.956 DEBUG:boto:Params: {} 2024-04-09T21:52:41.956 DEBUG:boto:Token: None 2024-04-09T21:52:41.957 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.957 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4= 2024-04-09T21:52:41.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4=', 'Content-Length': '0'} 2024-04-09T21:52:41.958 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef3d08d1e32bca3c-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:41.959 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:41.959 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:41.959 DEBUG:boto:path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.959 DEBUG:boto:auth_path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.959 DEBUG:boto:Method: GET 2024-04-09T21:52:41.959 DEBUG:boto:Path: /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.959 DEBUG:boto:Data: 2024-04-09T21:52:41.959 DEBUG:boto:Headers: {} 2024-04-09T21:52:41.959 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:41.960 DEBUG:boto:Port: 8000 2024-04-09T21:52:41.960 DEBUG:boto:Params: {} 2024-04-09T21:52:41.960 DEBUG:boto:Token: None 2024-04-09T21:52:41.960 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:41 GMT /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:41.960 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4= 2024-04-09T21:52:41.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4=', 'Content-Length': '0'} 2024-04-09T21:52:41.962 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009839b21786d1e8fa-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.006 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.007 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.007 DEBUG:boto:path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.007 DEBUG:boto:auth_path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.007 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.007 DEBUG:boto:Path: /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.007 DEBUG:boto:Data: 2024-04-09T21:52:42.007 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.007 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.007 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.007 DEBUG:boto:Params: {} 2024-04-09T21:52:42.008 DEBUG:boto:Token: None 2024-04-09T21:52:42.008 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.008 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4= 2024-04-09T21:52:42.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4=', 'Content-Length': '0'} 2024-04-09T21:52:42.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe98be0ec16c749f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.010 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.010 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.010 DEBUG:boto:path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.010 DEBUG:boto:auth_path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.010 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.011 DEBUG:boto:Path: /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.011 DEBUG:boto:Data: 2024-04-09T21:52:42.011 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.011 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.011 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.011 DEBUG:boto:Params: {} 2024-04-09T21:52:42.011 DEBUG:boto:Token: None 2024-04-09T21:52:42.011 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.011 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4= 2024-04-09T21:52:42.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4=', 'Content-Length': '0'} 2024-04-09T21:52:42.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fb8554a5ef025ff-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-04-09T21:52:42.013 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.013 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.013 DEBUG:boto:path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.013 DEBUG:boto:auth_path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.014 DEBUG:boto:Method: GET 2024-04-09T21:52:42.014 DEBUG:boto:Path: /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.014 DEBUG:boto:Data: 2024-04-09T21:52:42.014 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.014 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.014 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.014 DEBUG:boto:Params: {} 2024-04-09T21:52:42.014 DEBUG:boto:Token: None 2024-04-09T21:52:42.014 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.014 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww= 2024-04-09T21:52:42.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww=', 'Content-Length': '0'} 2024-04-09T21:52:42.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb92510717c616da-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.016 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.017 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.017 DEBUG:boto:path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.017 DEBUG:boto:auth_path=/xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.017 DEBUG:boto:Method: GET 2024-04-09T21:52:42.017 DEBUG:boto:Path: /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.017 DEBUG:boto:Data: 2024-04-09T21:52:42.017 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.017 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.017 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.017 DEBUG:boto:Params: {} 2024-04-09T21:52:42.018 DEBUG:boto:Token: None 2024-04-09T21:52:42.018 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28?versionId=null 2024-04-09T21:52:42.018 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww= 2024-04-09T21:52:42.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww=', 'Content-Length': '0'} 2024-04-09T21:52:42.019 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c54ba3a8bcb29b6-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.020 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.020 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.020 DEBUG:boto:path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.020 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.020 DEBUG:boto:Method: GET 2024-04-09T21:52:42.020 DEBUG:boto:Path: /xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.020 DEBUG:boto:Data: 2024-04-09T21:52:42.020 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.020 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.021 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.021 DEBUG:boto:Params: {} 2024-04-09T21:52:42.021 DEBUG:boto:Token: None 2024-04-09T21:52:42.021 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.021 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2188cfbab40cac7-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.024 DEBUG:boto:b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.024 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.024 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.024 DEBUG:boto:path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.024 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.024 DEBUG:boto:Method: GET 2024-04-09T21:52:42.025 DEBUG:boto:Path: /xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.025 DEBUG:boto:Data: 2024-04-09T21:52:42.025 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.025 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.025 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.025 DEBUG:boto:Params: {} 2024-04-09T21:52:42.025 DEBUG:boto:Token: None 2024-04-09T21:52:42.025 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.025 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdff60cf9a6042cc-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.028 DEBUG:boto:b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.028 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.028 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.028 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.028 DEBUG:boto:Path: /xryqqk-11/obj28 2024-04-09T21:52:42.028 DEBUG:boto:Data: 2024-04-09T21:52:42.029 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.029 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.029 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.029 DEBUG:boto:Params: {} 2024-04-09T21:52:42.029 DEBUG:boto:Token: None 2024-04-09T21:52:42.029 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28 2024-04-09T21:52:42.029 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw= 2024-04-09T21:52:42.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw=', 'Content-Length': '0'} 2024-04-09T21:52:42.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000781bd9547844d35e-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.031 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.031 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.031 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.031 DEBUG:boto:Path: /xryqqk-11/obj28 2024-04-09T21:52:42.031 DEBUG:boto:Data: 2024-04-09T21:52:42.031 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.031 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.032 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.032 DEBUG:boto:Params: {} 2024-04-09T21:52:42.032 DEBUG:boto:Token: None 2024-04-09T21:52:42.032 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28 2024-04-09T21:52:42.032 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw= 2024-04-09T21:52:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw=', 'Content-Length': '0'} 2024-04-09T21:52:42.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003378258b5bf1f4ea-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj28 2024-04-09T21:52:42.034 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.034 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.034 DEBUG:boto:Method: GET 2024-04-09T21:52:42.034 DEBUG:boto:Path: /xryqqk-11/obj28 2024-04-09T21:52:42.035 DEBUG:boto:Data: 2024-04-09T21:52:42.035 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.035 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.035 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.035 DEBUG:boto:Params: {} 2024-04-09T21:52:42.035 DEBUG:boto:Token: None 2024-04-09T21:52:42.035 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28 2024-04-09T21:52:42.035 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk= 2024-04-09T21:52:42.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk=', 'Content-Length': '0'} 2024-04-09T21:52:42.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c37cd9f365a1b23f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.078 DEBUG:boto:path=/xryqqk-11/obj28 2024-04-09T21:52:42.079 DEBUG:boto:auth_path=/xryqqk-11/obj28 2024-04-09T21:52:42.079 DEBUG:boto:Method: GET 2024-04-09T21:52:42.079 DEBUG:boto:Path: /xryqqk-11/obj28 2024-04-09T21:52:42.079 DEBUG:boto:Data: 2024-04-09T21:52:42.079 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.079 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.079 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.079 DEBUG:boto:Params: {} 2024-04-09T21:52:42.079 DEBUG:boto:Token: None 2024-04-09T21:52:42.080 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj28 2024-04-09T21:52:42.080 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk= 2024-04-09T21:52:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk=', 'Content-Length': '0'} 2024-04-09T21:52:42.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000765a649177994833-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.082 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.082 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.082 DEBUG:boto:path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.082 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.082 DEBUG:boto:Method: GET 2024-04-09T21:52:42.083 DEBUG:boto:Path: /xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.083 DEBUG:boto:Data: 2024-04-09T21:52:42.083 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.083 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.083 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.083 DEBUG:boto:Params: {} 2024-04-09T21:52:42.083 DEBUG:boto:Token: None 2024-04-09T21:52:42.083 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.083 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41de1123ad1fabb-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.086 DEBUG:boto:b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.086 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.086 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.086 DEBUG:boto:path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.087 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.087 DEBUG:boto:Method: GET 2024-04-09T21:52:42.087 DEBUG:boto:Path: /xryqqk-11/?prefix=obj28 2024-04-09T21:52:42.087 DEBUG:boto:Data: 2024-04-09T21:52:42.087 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.087 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.087 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.087 DEBUG:boto:Params: {} 2024-04-09T21:52:42.088 DEBUG:boto:Token: None 2024-04-09T21:52:42.088 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.088 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093c59ed3d768c866-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.091 DEBUG:boto:b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-09T21:52:42.091 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.092 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.092 DEBUG:boto:path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.092 DEBUG:boto:auth_path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.092 DEBUG:boto:Method: GET 2024-04-09T21:52:42.092 DEBUG:boto:Path: /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.092 DEBUG:boto:Data: 2024-04-09T21:52:42.092 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.092 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.092 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.092 DEBUG:boto:Params: {} 2024-04-09T21:52:42.092 DEBUG:boto:Token: None 2024-04-09T21:52:42.093 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.093 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-09T21:52:42.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-09T21:52:42.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004464579b1dd6f08d-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.094 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.095 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.095 DEBUG:boto:path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.095 DEBUG:boto:auth_path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.095 DEBUG:boto:Method: GET 2024-04-09T21:52:42.095 DEBUG:boto:Path: /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.095 DEBUG:boto:Data: 2024-04-09T21:52:42.095 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.095 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.095 DEBUG:boto:Params: {} 2024-04-09T21:52:42.096 DEBUG:boto:Token: None 2024-04-09T21:52:42.096 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.096 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-09T21:52:42.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-09T21:52:42.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000457645289995050d-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.142 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.143 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.143 DEBUG:boto:path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.143 DEBUG:boto:auth_path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.143 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.143 DEBUG:boto:Path: /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.143 DEBUG:boto:Data: 2024-04-09T21:52:42.143 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.143 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.143 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.143 DEBUG:boto:Params: {} 2024-04-09T21:52:42.144 DEBUG:boto:Token: None 2024-04-09T21:52:42.144 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc= 2024-04-09T21:52:42.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc=', 'Content-Length': '0'} 2024-04-09T21:52:42.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007193d9977d1a3cd-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.146 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.146 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.146 DEBUG:boto:path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.146 DEBUG:boto:auth_path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.146 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.146 DEBUG:boto:Path: /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.146 DEBUG:boto:Data: 2024-04-09T21:52:42.147 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.147 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.147 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.147 DEBUG:boto:Params: {} 2024-04-09T21:52:42.147 DEBUG:boto:Token: None 2024-04-09T21:52:42.147 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.147 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc= 2024-04-09T21:52:42.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc=', 'Content-Length': '0'} 2024-04-09T21:52:42.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008592d659c02416ec-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-09T21:52:42.149 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.149 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.150 DEBUG:boto:path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.150 DEBUG:boto:auth_path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.150 DEBUG:boto:Method: GET 2024-04-09T21:52:42.150 DEBUG:boto:Path: /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.150 DEBUG:boto:Data: 2024-04-09T21:52:42.150 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.150 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.150 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.150 DEBUG:boto:Params: {} 2024-04-09T21:52:42.151 DEBUG:boto:Token: None 2024-04-09T21:52:42.151 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.151 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-09T21:52:42.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-09T21:52:42.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000555e618a79336fb2-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.153 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.154 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.154 DEBUG:boto:path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.154 DEBUG:boto:auth_path=/xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.154 DEBUG:boto:Method: GET 2024-04-09T21:52:42.154 DEBUG:boto:Path: /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.154 DEBUG:boto:Data: 2024-04-09T21:52:42.154 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.154 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.154 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.154 DEBUG:boto:Params: {} 2024-04-09T21:52:42.155 DEBUG:boto:Token: None 2024-04-09T21:52:42.155 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29?versionId=null 2024-04-09T21:52:42.155 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-09T21:52:42.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-09T21:52:42.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b47794a65ba4f1ee-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.157 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.157 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.157 DEBUG:boto:path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.157 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.157 DEBUG:boto:Method: GET 2024-04-09T21:52:42.157 DEBUG:boto:Path: /xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.158 DEBUG:boto:Data: 2024-04-09T21:52:42.158 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.158 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.158 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.158 DEBUG:boto:Params: {} 2024-04-09T21:52:42.158 DEBUG:boto:Token: None 2024-04-09T21:52:42.158 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.158 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb3d420df8850f4b-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.160 DEBUG:boto:b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.161 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.161 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.161 DEBUG:boto:path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.161 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.161 DEBUG:boto:Method: GET 2024-04-09T21:52:42.161 DEBUG:boto:Path: /xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.162 DEBUG:boto:Data: 2024-04-09T21:52:42.162 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.162 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.162 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.162 DEBUG:boto:Params: {} 2024-04-09T21:52:42.162 DEBUG:boto:Token: None 2024-04-09T21:52:42.162 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.162 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138bbe9da8f007b9-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.165 DEBUG:boto:b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.166 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.166 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.166 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.166 DEBUG:boto:Path: /xryqqk-11/obj29 2024-04-09T21:52:42.166 DEBUG:boto:Data: 2024-04-09T21:52:42.167 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.167 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.167 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.167 DEBUG:boto:Params: {} 2024-04-09T21:52:42.168 DEBUG:boto:Token: None 2024-04-09T21:52:42.168 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29 2024-04-09T21:52:42.168 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs= 2024-04-09T21:52:42.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs=', 'Content-Length': '0'} 2024-04-09T21:52:42.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be6a8a068f704b88-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.170 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.170 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.170 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.170 DEBUG:boto:Path: /xryqqk-11/obj29 2024-04-09T21:52:42.170 DEBUG:boto:Data: 2024-04-09T21:52:42.170 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.170 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.171 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.171 DEBUG:boto:Params: {} 2024-04-09T21:52:42.171 DEBUG:boto:Token: None 2024-04-09T21:52:42.171 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29 2024-04-09T21:52:42.171 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs= 2024-04-09T21:52:42.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs=', 'Content-Length': '0'} 2024-04-09T21:52:42.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000570b919894da7f76-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-09T21:52:42.173 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.173 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.173 DEBUG:boto:Method: GET 2024-04-09T21:52:42.173 DEBUG:boto:Path: /xryqqk-11/obj29 2024-04-09T21:52:42.173 DEBUG:boto:Data: 2024-04-09T21:52:42.173 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.173 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.174 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.174 DEBUG:boto:Params: {} 2024-04-09T21:52:42.174 DEBUG:boto:Token: None 2024-04-09T21:52:42.174 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29 2024-04-09T21:52:42.174 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs= 2024-04-09T21:52:42.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs=', 'Content-Length': '0'} 2024-04-09T21:52:42.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb8b6daa8f6ec058-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.176 DEBUG:boto:path=/xryqqk-11/obj29 2024-04-09T21:52:42.176 DEBUG:boto:auth_path=/xryqqk-11/obj29 2024-04-09T21:52:42.176 DEBUG:boto:Method: GET 2024-04-09T21:52:42.176 DEBUG:boto:Path: /xryqqk-11/obj29 2024-04-09T21:52:42.176 DEBUG:boto:Data: 2024-04-09T21:52:42.176 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.176 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.176 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.177 DEBUG:boto:Params: {} 2024-04-09T21:52:42.177 DEBUG:boto:Token: None 2024-04-09T21:52:42.177 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj29 2024-04-09T21:52:42.177 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs= 2024-04-09T21:52:42.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs=', 'Content-Length': '0'} 2024-04-09T21:52:42.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000188494966e5c51cd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.179 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.179 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.179 DEBUG:boto:path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.179 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.179 DEBUG:boto:Method: GET 2024-04-09T21:52:42.179 DEBUG:boto:Path: /xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.179 DEBUG:boto:Data: 2024-04-09T21:52:42.180 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.180 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.180 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.180 DEBUG:boto:Params: {} 2024-04-09T21:52:42.180 DEBUG:boto:Token: None 2024-04-09T21:52:42.180 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.180 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4037aced0dc32ac-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.182 DEBUG:boto:b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.183 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.183 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.183 DEBUG:boto:path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.183 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.183 DEBUG:boto:Method: GET 2024-04-09T21:52:42.183 DEBUG:boto:Path: /xryqqk-11/?prefix=obj29 2024-04-09T21:52:42.184 DEBUG:boto:Data: 2024-04-09T21:52:42.184 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.184 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.184 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.184 DEBUG:boto:Params: {} 2024-04-09T21:52:42.184 DEBUG:boto:Token: None 2024-04-09T21:52:42.184 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.184 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d914a5f8d0e9d422-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.187 DEBUG:boto:b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-09T21:52:42.187 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.187 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.188 DEBUG:boto:path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.188 DEBUG:boto:auth_path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.188 DEBUG:boto:Method: GET 2024-04-09T21:52:42.188 DEBUG:boto:Path: /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.188 DEBUG:boto:Data: 2024-04-09T21:52:42.188 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.188 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.188 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.188 DEBUG:boto:Params: {} 2024-04-09T21:52:42.188 DEBUG:boto:Token: None 2024-04-09T21:52:42.188 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.189 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-09T21:52:42.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-09T21:52:42.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3613dd51cc2a3c1-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.190 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.191 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.191 DEBUG:boto:path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.191 DEBUG:boto:auth_path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.191 DEBUG:boto:Method: GET 2024-04-09T21:52:42.191 DEBUG:boto:Path: /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.191 DEBUG:boto:Data: 2024-04-09T21:52:42.191 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.191 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.191 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.191 DEBUG:boto:Params: {} 2024-04-09T21:52:42.191 DEBUG:boto:Token: None 2024-04-09T21:52:42.192 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.192 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-09T21:52:42.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-09T21:52:42.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c8a5450c926efd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.234 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.235 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.235 DEBUG:boto:path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.235 DEBUG:boto:auth_path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.235 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.235 DEBUG:boto:Path: /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.235 DEBUG:boto:Data: 2024-04-09T21:52:42.235 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.235 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.235 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.235 DEBUG:boto:Params: {} 2024-04-09T21:52:42.236 DEBUG:boto:Token: None 2024-04-09T21:52:42.236 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.236 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg= 2024-04-09T21:52:42.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg=', 'Content-Length': '0'} 2024-04-09T21:52:42.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036214a5caeadd0b4-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.238 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.238 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.238 DEBUG:boto:path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.238 DEBUG:boto:auth_path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.238 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.238 DEBUG:boto:Path: /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.238 DEBUG:boto:Data: 2024-04-09T21:52:42.238 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.238 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.238 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.239 DEBUG:boto:Params: {} 2024-04-09T21:52:42.239 DEBUG:boto:Token: None 2024-04-09T21:52:42.239 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.239 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg= 2024-04-09T21:52:42.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg=', 'Content-Length': '0'} 2024-04-09T21:52:42.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000213bf504c9b6c62f-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-09T21:52:42.241 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.241 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.241 DEBUG:boto:path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.241 DEBUG:boto:auth_path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.241 DEBUG:boto:Method: GET 2024-04-09T21:52:42.241 DEBUG:boto:Path: /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.241 DEBUG:boto:Data: 2024-04-09T21:52:42.241 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.242 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.242 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.242 DEBUG:boto:Params: {} 2024-04-09T21:52:42.242 DEBUG:boto:Token: None 2024-04-09T21:52:42.242 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.242 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-09T21:52:42.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-09T21:52:42.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027524190f554d28b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.244 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.244 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.244 DEBUG:boto:path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.244 DEBUG:boto:auth_path=/xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.245 DEBUG:boto:Method: GET 2024-04-09T21:52:42.245 DEBUG:boto:Path: /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.245 DEBUG:boto:Data: 2024-04-09T21:52:42.245 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.245 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.245 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.245 DEBUG:boto:Params: {} 2024-04-09T21:52:42.245 DEBUG:boto:Token: None 2024-04-09T21:52:42.245 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3?versionId=null 2024-04-09T21:52:42.246 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-09T21:52:42.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-09T21:52:42.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005856995e59851206-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.248 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.248 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.248 DEBUG:boto:path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.248 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.248 DEBUG:boto:Method: GET 2024-04-09T21:52:42.248 DEBUG:boto:Path: /xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.248 DEBUG:boto:Data: 2024-04-09T21:52:42.248 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.248 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.249 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.249 DEBUG:boto:Params: {} 2024-04-09T21:52:42.249 DEBUG:boto:Token: None 2024-04-09T21:52:42.249 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.249 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1677678598afd8e-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.251 DEBUG:boto:b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.252 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.252 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.252 DEBUG:boto:path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.252 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.253 DEBUG:boto:Method: GET 2024-04-09T21:52:42.253 DEBUG:boto:Path: /xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.253 DEBUG:boto:Data: 2024-04-09T21:52:42.253 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.253 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.253 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.253 DEBUG:boto:Params: {} 2024-04-09T21:52:42.253 DEBUG:boto:Token: None 2024-04-09T21:52:42.253 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.253 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbce69e769dee13f-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.256 DEBUG:boto:b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.256 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.257 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.257 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.257 DEBUG:boto:Path: /xryqqk-11/obj3 2024-04-09T21:52:42.257 DEBUG:boto:Data: 2024-04-09T21:52:42.257 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.257 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.257 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.257 DEBUG:boto:Params: {} 2024-04-09T21:52:42.257 DEBUG:boto:Token: None 2024-04-09T21:52:42.257 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3 2024-04-09T21:52:42.258 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA= 2024-04-09T21:52:42.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA=', 'Content-Length': '0'} 2024-04-09T21:52:42.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acb3d0af6fc6df91-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.259 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.260 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.260 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.260 DEBUG:boto:Path: /xryqqk-11/obj3 2024-04-09T21:52:42.260 DEBUG:boto:Data: 2024-04-09T21:52:42.260 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.261 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.261 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.261 DEBUG:boto:Params: {} 2024-04-09T21:52:42.261 DEBUG:boto:Token: None 2024-04-09T21:52:42.262 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3 2024-04-09T21:52:42.262 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA= 2024-04-09T21:52:42.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA=', 'Content-Length': '0'} 2024-04-09T21:52:42.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acf845960b629df8-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-09T21:52:42.264 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.265 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.265 DEBUG:boto:Method: GET 2024-04-09T21:52:42.265 DEBUG:boto:Path: /xryqqk-11/obj3 2024-04-09T21:52:42.265 DEBUG:boto:Data: 2024-04-09T21:52:42.265 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.265 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.265 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.265 DEBUG:boto:Params: {} 2024-04-09T21:52:42.265 DEBUG:boto:Token: None 2024-04-09T21:52:42.266 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3 2024-04-09T21:52:42.266 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8= 2024-04-09T21:52:42.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8=', 'Content-Length': '0'} 2024-04-09T21:52:42.267 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af17a84d37f8ce0b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.310 DEBUG:boto:path=/xryqqk-11/obj3 2024-04-09T21:52:42.311 DEBUG:boto:auth_path=/xryqqk-11/obj3 2024-04-09T21:52:42.311 DEBUG:boto:Method: GET 2024-04-09T21:52:42.311 DEBUG:boto:Path: /xryqqk-11/obj3 2024-04-09T21:52:42.311 DEBUG:boto:Data: 2024-04-09T21:52:42.311 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.311 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.311 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.311 DEBUG:boto:Params: {} 2024-04-09T21:52:42.311 DEBUG:boto:Token: None 2024-04-09T21:52:42.312 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj3 2024-04-09T21:52:42.312 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8= 2024-04-09T21:52:42.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8=', 'Content-Length': '0'} 2024-04-09T21:52:42.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bab5071c9b10bfd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.314 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.314 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.314 DEBUG:boto:path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.314 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.314 DEBUG:boto:Method: GET 2024-04-09T21:52:42.314 DEBUG:boto:Path: /xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.315 DEBUG:boto:Data: 2024-04-09T21:52:42.315 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.315 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.315 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.315 DEBUG:boto:Params: {} 2024-04-09T21:52:42.315 DEBUG:boto:Token: None 2024-04-09T21:52:42.315 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.315 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03dcfbd932fcdd6-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.318 DEBUG:boto:b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.319 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.319 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.319 DEBUG:boto:path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.319 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.319 DEBUG:boto:Method: GET 2024-04-09T21:52:42.319 DEBUG:boto:Path: /xryqqk-11/?prefix=obj3 2024-04-09T21:52:42.319 DEBUG:boto:Data: 2024-04-09T21:52:42.320 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.320 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.320 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.320 DEBUG:boto:Params: {} 2024-04-09T21:52:42.320 DEBUG:boto:Token: None 2024-04-09T21:52:42.320 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ed797f01d14196f-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.322 DEBUG:boto:b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-04-09T21:52:42.324 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.324 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.324 DEBUG:boto:path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.324 DEBUG:boto:auth_path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.324 DEBUG:boto:Method: GET 2024-04-09T21:52:42.324 DEBUG:boto:Path: /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.324 DEBUG:boto:Data: 2024-04-09T21:52:42.324 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.324 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.325 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.325 DEBUG:boto:Params: {} 2024-04-09T21:52:42.325 DEBUG:boto:Token: None 2024-04-09T21:52:42.325 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.325 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-09T21:52:42.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-09T21:52:42.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055d4131e551780f3-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.327 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.327 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.327 DEBUG:boto:path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.327 DEBUG:boto:auth_path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.328 DEBUG:boto:Method: GET 2024-04-09T21:52:42.328 DEBUG:boto:Path: /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.328 DEBUG:boto:Data: 2024-04-09T21:52:42.328 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.328 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.328 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.328 DEBUG:boto:Params: {} 2024-04-09T21:52:42.328 DEBUG:boto:Token: None 2024-04-09T21:52:42.328 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.329 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-09T21:52:42.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-09T21:52:42.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dffddcb2a754dafd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.378 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.379 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.379 DEBUG:boto:path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.379 DEBUG:boto:auth_path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.379 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.379 DEBUG:boto:Path: /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.379 DEBUG:boto:Data: 2024-04-09T21:52:42.379 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.379 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.379 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.379 DEBUG:boto:Params: {} 2024-04-09T21:52:42.380 DEBUG:boto:Token: None 2024-04-09T21:52:42.380 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.380 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic= 2024-04-09T21:52:42.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic=', 'Content-Length': '0'} 2024-04-09T21:52:42.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dc68089cf297ebc-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.382 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.382 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.382 DEBUG:boto:path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.383 DEBUG:boto:auth_path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.383 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.383 DEBUG:boto:Path: /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.383 DEBUG:boto:Data: 2024-04-09T21:52:42.383 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.383 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.383 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.383 DEBUG:boto:Params: {} 2024-04-09T21:52:42.383 DEBUG:boto:Token: None 2024-04-09T21:52:42.384 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.384 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic= 2024-04-09T21:52:42.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic=', 'Content-Length': '0'} 2024-04-09T21:52:42.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b34a7353ec8c1760-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-04-09T21:52:42.386 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.386 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.386 DEBUG:boto:path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.386 DEBUG:boto:auth_path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.386 DEBUG:boto:Method: GET 2024-04-09T21:52:42.386 DEBUG:boto:Path: /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.387 DEBUG:boto:Data: 2024-04-09T21:52:42.387 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.387 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.387 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.387 DEBUG:boto:Params: {} 2024-04-09T21:52:42.387 DEBUG:boto:Token: None 2024-04-09T21:52:42.387 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.387 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-09T21:52:42.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-09T21:52:42.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a12d3c243cdc80eb-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.389 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.389 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.389 DEBUG:boto:path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.390 DEBUG:boto:auth_path=/xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.390 DEBUG:boto:Method: GET 2024-04-09T21:52:42.390 DEBUG:boto:Path: /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.390 DEBUG:boto:Data: 2024-04-09T21:52:42.390 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.390 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.390 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.390 DEBUG:boto:Params: {} 2024-04-09T21:52:42.390 DEBUG:boto:Token: None 2024-04-09T21:52:42.391 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30?versionId=null 2024-04-09T21:52:42.391 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-09T21:52:42.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-09T21:52:42.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abda4e17631356c2-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.393 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.393 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.393 DEBUG:boto:path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.393 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.393 DEBUG:boto:Method: GET 2024-04-09T21:52:42.393 DEBUG:boto:Path: /xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.393 DEBUG:boto:Data: 2024-04-09T21:52:42.393 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.393 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.394 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.394 DEBUG:boto:Params: {} 2024-04-09T21:52:42.394 DEBUG:boto:Token: None 2024-04-09T21:52:42.394 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.394 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1825f9be69ee84d-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.396 DEBUG:boto:b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.397 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.397 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.397 DEBUG:boto:path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.397 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.397 DEBUG:boto:Method: GET 2024-04-09T21:52:42.397 DEBUG:boto:Path: /xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.397 DEBUG:boto:Data: 2024-04-09T21:52:42.397 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.398 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.398 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.398 DEBUG:boto:Params: {} 2024-04-09T21:52:42.398 DEBUG:boto:Token: None 2024-04-09T21:52:42.398 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.398 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c79efe941044544-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.402 DEBUG:boto:b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.402 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.402 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.402 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.402 DEBUG:boto:Path: /xryqqk-11/obj30 2024-04-09T21:52:42.402 DEBUG:boto:Data: 2024-04-09T21:52:42.403 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.403 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.403 DEBUG:boto:Params: {} 2024-04-09T21:52:42.403 DEBUG:boto:Token: None 2024-04-09T21:52:42.403 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30 2024-04-09T21:52:42.403 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg= 2024-04-09T21:52:42.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg=', 'Content-Length': '0'} 2024-04-09T21:52:42.405 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070c8db4a62483ef7-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.405 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.405 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.405 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.405 DEBUG:boto:Path: /xryqqk-11/obj30 2024-04-09T21:52:42.406 DEBUG:boto:Data: 2024-04-09T21:52:42.406 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.406 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.406 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.406 DEBUG:boto:Params: {} 2024-04-09T21:52:42.406 DEBUG:boto:Token: None 2024-04-09T21:52:42.406 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30 2024-04-09T21:52:42.406 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg= 2024-04-09T21:52:42.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg=', 'Content-Length': '0'} 2024-04-09T21:52:42.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005859f0a71026667e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj30 2024-04-09T21:52:42.408 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.408 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.409 DEBUG:boto:Method: GET 2024-04-09T21:52:42.409 DEBUG:boto:Path: /xryqqk-11/obj30 2024-04-09T21:52:42.409 DEBUG:boto:Data: 2024-04-09T21:52:42.409 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.409 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.409 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.409 DEBUG:boto:Params: {} 2024-04-09T21:52:42.409 DEBUG:boto:Token: None 2024-04-09T21:52:42.409 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30 2024-04-09T21:52:42.409 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY= 2024-04-09T21:52:42.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY=', 'Content-Length': '0'} 2024-04-09T21:52:42.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb05f6261cb1a7a8-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.411 DEBUG:boto:path=/xryqqk-11/obj30 2024-04-09T21:52:42.412 DEBUG:boto:auth_path=/xryqqk-11/obj30 2024-04-09T21:52:42.412 DEBUG:boto:Method: GET 2024-04-09T21:52:42.412 DEBUG:boto:Path: /xryqqk-11/obj30 2024-04-09T21:52:42.412 DEBUG:boto:Data: 2024-04-09T21:52:42.412 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.412 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.412 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.412 DEBUG:boto:Params: {} 2024-04-09T21:52:42.412 DEBUG:boto:Token: None 2024-04-09T21:52:42.413 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj30 2024-04-09T21:52:42.413 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY= 2024-04-09T21:52:42.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY=', 'Content-Length': '0'} 2024-04-09T21:52:42.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000744f15ed97169eb2-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.414 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.415 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.415 DEBUG:boto:path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.415 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.415 DEBUG:boto:Method: GET 2024-04-09T21:52:42.415 DEBUG:boto:Path: /xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.415 DEBUG:boto:Data: 2024-04-09T21:52:42.415 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.415 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.415 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.415 DEBUG:boto:Params: {} 2024-04-09T21:52:42.416 DEBUG:boto:Token: None 2024-04-09T21:52:42.416 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.416 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e2a2c162cd1c0a-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.418 DEBUG:boto:b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.419 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.419 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.419 DEBUG:boto:path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.419 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.419 DEBUG:boto:Method: GET 2024-04-09T21:52:42.419 DEBUG:boto:Path: /xryqqk-11/?prefix=obj30 2024-04-09T21:52:42.419 DEBUG:boto:Data: 2024-04-09T21:52:42.419 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.419 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.420 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.420 DEBUG:boto:Params: {} 2024-04-09T21:52:42.420 DEBUG:boto:Token: None 2024-04-09T21:52:42.420 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.420 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058bd0867e1f307ae-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.423 DEBUG:boto:b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-09T21:52:42.423 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.423 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.423 DEBUG:boto:path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.424 DEBUG:boto:auth_path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.424 DEBUG:boto:Method: GET 2024-04-09T21:52:42.424 DEBUG:boto:Path: /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.424 DEBUG:boto:Data: 2024-04-09T21:52:42.424 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.424 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.424 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.424 DEBUG:boto:Params: {} 2024-04-09T21:52:42.424 DEBUG:boto:Token: None 2024-04-09T21:52:42.424 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.425 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-09T21:52:42.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-09T21:52:42.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000330c2dbdfceeade-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.428 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.428 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.428 DEBUG:boto:path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.428 DEBUG:boto:auth_path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.428 DEBUG:boto:Method: GET 2024-04-09T21:52:42.428 DEBUG:boto:Path: /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.428 DEBUG:boto:Data: 2024-04-09T21:52:42.428 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.428 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.428 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.428 DEBUG:boto:Params: {} 2024-04-09T21:52:42.428 DEBUG:boto:Token: None 2024-04-09T21:52:42.429 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.429 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-09T21:52:42.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-09T21:52:42.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007451981b606dca01-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.478 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.482 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.482 DEBUG:boto:path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.482 DEBUG:boto:auth_path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.482 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.483 DEBUG:boto:Path: /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.483 DEBUG:boto:Data: 2024-04-09T21:52:42.483 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.483 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.483 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.483 DEBUG:boto:Params: {} 2024-04-09T21:52:42.483 DEBUG:boto:Token: None 2024-04-09T21:52:42.483 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.484 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY= 2024-04-09T21:52:42.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY=', 'Content-Length': '0'} 2024-04-09T21:52:42.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000512af2107049c41f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.486 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.486 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.486 DEBUG:boto:path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.486 DEBUG:boto:auth_path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.486 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.487 DEBUG:boto:Path: /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.487 DEBUG:boto:Data: 2024-04-09T21:52:42.487 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.487 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.487 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.487 DEBUG:boto:Params: {} 2024-04-09T21:52:42.487 DEBUG:boto:Token: None 2024-04-09T21:52:42.487 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.487 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY= 2024-04-09T21:52:42.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY=', 'Content-Length': '0'} 2024-04-09T21:52:42.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e9097a87368c9dc-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-09T21:52:42.489 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.489 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.489 DEBUG:boto:path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.490 DEBUG:boto:auth_path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.490 DEBUG:boto:Method: GET 2024-04-09T21:52:42.490 DEBUG:boto:Path: /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.490 DEBUG:boto:Data: 2024-04-09T21:52:42.490 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.490 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.490 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.490 DEBUG:boto:Params: {} 2024-04-09T21:52:42.491 DEBUG:boto:Token: None 2024-04-09T21:52:42.491 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.491 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-09T21:52:42.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-09T21:52:42.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069051853f6edb83f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.494 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.494 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.494 DEBUG:boto:path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.494 DEBUG:boto:auth_path=/xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.494 DEBUG:boto:Method: GET 2024-04-09T21:52:42.494 DEBUG:boto:Path: /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.494 DEBUG:boto:Data: 2024-04-09T21:52:42.494 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.494 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.494 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.494 DEBUG:boto:Params: {} 2024-04-09T21:52:42.495 DEBUG:boto:Token: None 2024-04-09T21:52:42.495 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31?versionId=null 2024-04-09T21:52:42.495 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-09T21:52:42.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-09T21:52:42.497 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000192626bfcadbcef2-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.497 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.497 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.497 DEBUG:boto:path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.497 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.497 DEBUG:boto:Method: GET 2024-04-09T21:52:42.497 DEBUG:boto:Path: /xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.498 DEBUG:boto:Data: 2024-04-09T21:52:42.498 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.498 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.498 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.498 DEBUG:boto:Params: {} 2024-04-09T21:52:42.498 DEBUG:boto:Token: None 2024-04-09T21:52:42.498 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.498 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000121a3a6ea77bc3a8-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.501 DEBUG:boto:b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.501 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.501 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.501 DEBUG:boto:path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.501 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.502 DEBUG:boto:Method: GET 2024-04-09T21:52:42.502 DEBUG:boto:Path: /xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.502 DEBUG:boto:Data: 2024-04-09T21:52:42.502 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.502 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.502 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.502 DEBUG:boto:Params: {} 2024-04-09T21:52:42.502 DEBUG:boto:Token: None 2024-04-09T21:52:42.502 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.503 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001710d2cec6bd3625-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.505 DEBUG:boto:b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.505 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.505 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.505 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.505 DEBUG:boto:Path: /xryqqk-11/obj31 2024-04-09T21:52:42.506 DEBUG:boto:Data: 2024-04-09T21:52:42.506 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.506 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.506 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.506 DEBUG:boto:Params: {} 2024-04-09T21:52:42.506 DEBUG:boto:Token: None 2024-04-09T21:52:42.506 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31 2024-04-09T21:52:42.506 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg= 2024-04-09T21:52:42.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg=', 'Content-Length': '0'} 2024-04-09T21:52:42.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000629e517905762560-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.508 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.508 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.508 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.508 DEBUG:boto:Path: /xryqqk-11/obj31 2024-04-09T21:52:42.508 DEBUG:boto:Data: 2024-04-09T21:52:42.508 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.509 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.509 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.509 DEBUG:boto:Params: {} 2024-04-09T21:52:42.509 DEBUG:boto:Token: None 2024-04-09T21:52:42.509 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31 2024-04-09T21:52:42.509 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg= 2024-04-09T21:52:42.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg=', 'Content-Length': '0'} 2024-04-09T21:52:42.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000829284b538454ccb-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-09T21:52:42.511 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.511 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.511 DEBUG:boto:Method: GET 2024-04-09T21:52:42.511 DEBUG:boto:Path: /xryqqk-11/obj31 2024-04-09T21:52:42.511 DEBUG:boto:Data: 2024-04-09T21:52:42.512 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.512 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.512 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.512 DEBUG:boto:Params: {} 2024-04-09T21:52:42.512 DEBUG:boto:Token: None 2024-04-09T21:52:42.512 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31 2024-04-09T21:52:42.512 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go= 2024-04-09T21:52:42.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go=', 'Content-Length': '0'} 2024-04-09T21:52:42.514 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947871d58d138c0c-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.514 DEBUG:boto:path=/xryqqk-11/obj31 2024-04-09T21:52:42.514 DEBUG:boto:auth_path=/xryqqk-11/obj31 2024-04-09T21:52:42.514 DEBUG:boto:Method: GET 2024-04-09T21:52:42.514 DEBUG:boto:Path: /xryqqk-11/obj31 2024-04-09T21:52:42.514 DEBUG:boto:Data: 2024-04-09T21:52:42.515 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.515 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.515 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.515 DEBUG:boto:Params: {} 2024-04-09T21:52:42.515 DEBUG:boto:Token: None 2024-04-09T21:52:42.515 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj31 2024-04-09T21:52:42.515 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go= 2024-04-09T21:52:42.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go=', 'Content-Length': '0'} 2024-04-09T21:52:42.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f2ba00ef5b0c08-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.517 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.517 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.517 DEBUG:boto:path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.517 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.518 DEBUG:boto:Method: GET 2024-04-09T21:52:42.518 DEBUG:boto:Path: /xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.518 DEBUG:boto:Data: 2024-04-09T21:52:42.518 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.518 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.518 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.518 DEBUG:boto:Params: {} 2024-04-09T21:52:42.518 DEBUG:boto:Token: None 2024-04-09T21:52:42.518 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.518 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f549495567bd580-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.521 DEBUG:boto:b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.521 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.521 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.521 DEBUG:boto:path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.521 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.521 DEBUG:boto:Method: GET 2024-04-09T21:52:42.522 DEBUG:boto:Path: /xryqqk-11/?prefix=obj31 2024-04-09T21:52:42.522 DEBUG:boto:Data: 2024-04-09T21:52:42.522 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.522 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.522 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.522 DEBUG:boto:Params: {} 2024-04-09T21:52:42.522 DEBUG:boto:Token: None 2024-04-09T21:52:42.522 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.522 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a23334ad1a143e-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.525 DEBUG:boto:b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-04-09T21:52:42.526 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.526 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.526 DEBUG:boto:path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.526 DEBUG:boto:auth_path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.526 DEBUG:boto:Method: GET 2024-04-09T21:52:42.526 DEBUG:boto:Path: /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.526 DEBUG:boto:Data: 2024-04-09T21:52:42.526 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.526 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.526 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.527 DEBUG:boto:Params: {} 2024-04-09T21:52:42.527 DEBUG:boto:Token: None 2024-04-09T21:52:42.527 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.527 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-09T21:52:42.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-09T21:52:42.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ffa179b985fb77c-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.529 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.529 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.529 DEBUG:boto:path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.529 DEBUG:boto:auth_path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.529 DEBUG:boto:Method: GET 2024-04-09T21:52:42.529 DEBUG:boto:Path: /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.530 DEBUG:boto:Data: 2024-04-09T21:52:42.530 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.530 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.530 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.530 DEBUG:boto:Params: {} 2024-04-09T21:52:42.530 DEBUG:boto:Token: None 2024-04-09T21:52:42.530 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.530 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-09T21:52:42.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-09T21:52:42.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfee6f6bbdf3ef67-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.532 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.532 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.532 DEBUG:boto:path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.533 DEBUG:boto:auth_path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.533 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.533 DEBUG:boto:Path: /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.533 DEBUG:boto:Data: 2024-04-09T21:52:42.533 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.533 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.533 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.533 DEBUG:boto:Params: {} 2024-04-09T21:52:42.533 DEBUG:boto:Token: None 2024-04-09T21:52:42.533 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.534 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM= 2024-04-09T21:52:42.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM=', 'Content-Length': '0'} 2024-04-09T21:52:42.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000966e48b6eb190fbe-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.535 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.536 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.536 DEBUG:boto:path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.536 DEBUG:boto:auth_path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.536 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.536 DEBUG:boto:Path: /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.536 DEBUG:boto:Data: 2024-04-09T21:52:42.536 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.536 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.536 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.536 DEBUG:boto:Params: {} 2024-04-09T21:52:42.536 DEBUG:boto:Token: None 2024-04-09T21:52:42.537 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.537 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM= 2024-04-09T21:52:42.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM=', 'Content-Length': '0'} 2024-04-09T21:52:42.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e286148a17643c0-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-04-09T21:52:42.538 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.539 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.539 DEBUG:boto:path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.539 DEBUG:boto:auth_path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.539 DEBUG:boto:Method: GET 2024-04-09T21:52:42.539 DEBUG:boto:Path: /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.539 DEBUG:boto:Data: 2024-04-09T21:52:42.539 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.539 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.539 DEBUG:boto:Params: {} 2024-04-09T21:52:42.539 DEBUG:boto:Token: None 2024-04-09T21:52:42.540 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-09T21:52:42.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-09T21:52:42.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6ce18a3ce31d41b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.541 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.542 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.542 DEBUG:boto:path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.542 DEBUG:boto:auth_path=/xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.542 DEBUG:boto:Method: GET 2024-04-09T21:52:42.542 DEBUG:boto:Path: /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.542 DEBUG:boto:Data: 2024-04-09T21:52:42.542 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.542 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.542 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.542 DEBUG:boto:Params: {} 2024-04-09T21:52:42.542 DEBUG:boto:Token: None 2024-04-09T21:52:42.543 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32?versionId=null 2024-04-09T21:52:42.543 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-09T21:52:42.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-09T21:52:42.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f7b5f5fdba3fdf-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.544 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.545 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.545 DEBUG:boto:path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.545 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.545 DEBUG:boto:Method: GET 2024-04-09T21:52:42.545 DEBUG:boto:Path: /xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.545 DEBUG:boto:Data: 2024-04-09T21:52:42.545 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.545 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.545 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.545 DEBUG:boto:Params: {} 2024-04-09T21:52:42.545 DEBUG:boto:Token: None 2024-04-09T21:52:42.546 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.546 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd53d7a5a19a1df8-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.548 DEBUG:boto:b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.548 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.548 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.548 DEBUG:boto:path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.548 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.548 DEBUG:boto:Method: GET 2024-04-09T21:52:42.549 DEBUG:boto:Path: /xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.549 DEBUG:boto:Data: 2024-04-09T21:52:42.549 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.549 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.549 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.549 DEBUG:boto:Params: {} 2024-04-09T21:52:42.549 DEBUG:boto:Token: None 2024-04-09T21:52:42.549 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.549 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b32250b257c3a8-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.551 DEBUG:boto:b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.552 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.552 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.552 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.552 DEBUG:boto:Path: /xryqqk-11/obj32 2024-04-09T21:52:42.552 DEBUG:boto:Data: 2024-04-09T21:52:42.552 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.552 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.552 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.552 DEBUG:boto:Params: {} 2024-04-09T21:52:42.552 DEBUG:boto:Token: None 2024-04-09T21:52:42.553 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32 2024-04-09T21:52:42.553 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I= 2024-04-09T21:52:42.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I=', 'Content-Length': '0'} 2024-04-09T21:52:42.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9f27c5c95beb848-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.554 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.554 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.555 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.555 DEBUG:boto:Path: /xryqqk-11/obj32 2024-04-09T21:52:42.555 DEBUG:boto:Data: 2024-04-09T21:52:42.555 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.555 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.555 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.555 DEBUG:boto:Params: {} 2024-04-09T21:52:42.555 DEBUG:boto:Token: None 2024-04-09T21:52:42.555 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32 2024-04-09T21:52:42.555 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I= 2024-04-09T21:52:42.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I=', 'Content-Length': '0'} 2024-04-09T21:52:42.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c2443c09152388-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj32 2024-04-09T21:52:42.557 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.557 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.558 DEBUG:boto:Method: GET 2024-04-09T21:52:42.558 DEBUG:boto:Path: /xryqqk-11/obj32 2024-04-09T21:52:42.558 DEBUG:boto:Data: 2024-04-09T21:52:42.558 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.558 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.558 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.558 DEBUG:boto:Params: {} 2024-04-09T21:52:42.558 DEBUG:boto:Token: None 2024-04-09T21:52:42.558 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32 2024-04-09T21:52:42.558 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs= 2024-04-09T21:52:42.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs=', 'Content-Length': '0'} 2024-04-09T21:52:42.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5483e64bed333de-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.560 DEBUG:boto:path=/xryqqk-11/obj32 2024-04-09T21:52:42.560 DEBUG:boto:auth_path=/xryqqk-11/obj32 2024-04-09T21:52:42.560 DEBUG:boto:Method: GET 2024-04-09T21:52:42.560 DEBUG:boto:Path: /xryqqk-11/obj32 2024-04-09T21:52:42.561 DEBUG:boto:Data: 2024-04-09T21:52:42.561 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.561 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.561 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.561 DEBUG:boto:Params: {} 2024-04-09T21:52:42.561 DEBUG:boto:Token: None 2024-04-09T21:52:42.561 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj32 2024-04-09T21:52:42.561 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs= 2024-04-09T21:52:42.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs=', 'Content-Length': '0'} 2024-04-09T21:52:42.563 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004686564320fa874d-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.610 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.611 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.611 DEBUG:boto:path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.611 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.611 DEBUG:boto:Method: GET 2024-04-09T21:52:42.611 DEBUG:boto:Path: /xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.611 DEBUG:boto:Data: 2024-04-09T21:52:42.611 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.611 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.611 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.612 DEBUG:boto:Params: {} 2024-04-09T21:52:42.612 DEBUG:boto:Token: None 2024-04-09T21:52:42.612 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.612 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224eef154f29818e-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.614 DEBUG:boto:b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.615 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.615 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.615 DEBUG:boto:path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.615 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.615 DEBUG:boto:Method: GET 2024-04-09T21:52:42.615 DEBUG:boto:Path: /xryqqk-11/?prefix=obj32 2024-04-09T21:52:42.615 DEBUG:boto:Data: 2024-04-09T21:52:42.616 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.616 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.616 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.616 DEBUG:boto:Params: {} 2024-04-09T21:52:42.616 DEBUG:boto:Token: None 2024-04-09T21:52:42.616 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.616 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e4d28339de9850-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.618 DEBUG:boto:b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-09T21:52:42.619 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.619 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.619 DEBUG:boto:path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.619 DEBUG:boto:auth_path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.619 DEBUG:boto:Method: GET 2024-04-09T21:52:42.620 DEBUG:boto:Path: /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.620 DEBUG:boto:Data: 2024-04-09T21:52:42.620 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.620 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.620 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.620 DEBUG:boto:Params: {} 2024-04-09T21:52:42.620 DEBUG:boto:Token: None 2024-04-09T21:52:42.620 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.620 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-09T21:52:42.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-09T21:52:42.622 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3f15539e261c478-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.662 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.662 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.663 DEBUG:boto:path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.663 DEBUG:boto:auth_path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.663 DEBUG:boto:Method: GET 2024-04-09T21:52:42.663 DEBUG:boto:Path: /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.663 DEBUG:boto:Data: 2024-04-09T21:52:42.663 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.663 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.663 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.663 DEBUG:boto:Params: {} 2024-04-09T21:52:42.663 DEBUG:boto:Token: None 2024-04-09T21:52:42.664 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-09T21:52:42.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-09T21:52:42.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056ce783c3f3716a0-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.666 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.666 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.666 DEBUG:boto:path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.666 DEBUG:boto:auth_path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.666 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.666 DEBUG:boto:Path: /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.666 DEBUG:boto:Data: 2024-04-09T21:52:42.666 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.667 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.667 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.667 DEBUG:boto:Params: {} 2024-04-09T21:52:42.667 DEBUG:boto:Token: None 2024-04-09T21:52:42.667 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.667 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s= 2024-04-09T21:52:42.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s=', 'Content-Length': '0'} 2024-04-09T21:52:42.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da797f84d3635c08-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.669 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.669 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.669 DEBUG:boto:path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.669 DEBUG:boto:auth_path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.669 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.670 DEBUG:boto:Path: /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.670 DEBUG:boto:Data: 2024-04-09T21:52:42.670 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.670 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.670 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.670 DEBUG:boto:Params: {} 2024-04-09T21:52:42.670 DEBUG:boto:Token: None 2024-04-09T21:52:42.670 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.670 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s= 2024-04-09T21:52:42.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s=', 'Content-Length': '0'} 2024-04-09T21:52:42.672 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053eb32b41cf020b4-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-09T21:52:42.672 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.672 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.673 DEBUG:boto:path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.673 DEBUG:boto:auth_path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.673 DEBUG:boto:Method: GET 2024-04-09T21:52:42.673 DEBUG:boto:Path: /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.673 DEBUG:boto:Data: 2024-04-09T21:52:42.673 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.673 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.673 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.673 DEBUG:boto:Params: {} 2024-04-09T21:52:42.673 DEBUG:boto:Token: None 2024-04-09T21:52:42.673 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.674 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-09T21:52:42.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-09T21:52:42.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8f62a6a4ee041bb-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.676 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.676 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.676 DEBUG:boto:path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.676 DEBUG:boto:auth_path=/xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.676 DEBUG:boto:Method: GET 2024-04-09T21:52:42.676 DEBUG:boto:Path: /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.676 DEBUG:boto:Data: 2024-04-09T21:52:42.676 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.676 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.676 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.677 DEBUG:boto:Params: {} 2024-04-09T21:52:42.677 DEBUG:boto:Token: None 2024-04-09T21:52:42.677 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33?versionId=null 2024-04-09T21:52:42.677 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-09T21:52:42.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-09T21:52:42.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f34e5c98ee0cdaee-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.681 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.681 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.681 DEBUG:boto:path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.682 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.682 DEBUG:boto:Method: GET 2024-04-09T21:52:42.682 DEBUG:boto:Path: /xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.682 DEBUG:boto:Data: 2024-04-09T21:52:42.682 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.682 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.682 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.682 DEBUG:boto:Params: {} 2024-04-09T21:52:42.682 DEBUG:boto:Token: None 2024-04-09T21:52:42.682 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.683 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097d17f9416da040c-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.685 DEBUG:boto:b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.686 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.686 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.686 DEBUG:boto:path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.686 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.687 DEBUG:boto:Method: GET 2024-04-09T21:52:42.687 DEBUG:boto:Path: /xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.687 DEBUG:boto:Data: 2024-04-09T21:52:42.687 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.687 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.687 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.687 DEBUG:boto:Params: {} 2024-04-09T21:52:42.687 DEBUG:boto:Token: None 2024-04-09T21:52:42.687 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.687 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c5c50a61000be9-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.690 DEBUG:boto:b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.690 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.690 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.690 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.690 DEBUG:boto:Path: /xryqqk-11/obj33 2024-04-09T21:52:42.690 DEBUG:boto:Data: 2024-04-09T21:52:42.690 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.690 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.691 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.691 DEBUG:boto:Params: {} 2024-04-09T21:52:42.691 DEBUG:boto:Token: None 2024-04-09T21:52:42.691 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33 2024-04-09T21:52:42.691 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o= 2024-04-09T21:52:42.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o=', 'Content-Length': '0'} 2024-04-09T21:52:42.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000800d889970d2161d-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.693 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.693 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.693 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.693 DEBUG:boto:Path: /xryqqk-11/obj33 2024-04-09T21:52:42.693 DEBUG:boto:Data: 2024-04-09T21:52:42.693 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.693 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.693 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.693 DEBUG:boto:Params: {} 2024-04-09T21:52:42.694 DEBUG:boto:Token: None 2024-04-09T21:52:42.694 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33 2024-04-09T21:52:42.694 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o= 2024-04-09T21:52:42.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o=', 'Content-Length': '0'} 2024-04-09T21:52:42.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009aadbc309d87e35e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-09T21:52:42.696 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.696 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.696 DEBUG:boto:Method: GET 2024-04-09T21:52:42.696 DEBUG:boto:Path: /xryqqk-11/obj33 2024-04-09T21:52:42.696 DEBUG:boto:Data: 2024-04-09T21:52:42.696 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.696 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.696 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.696 DEBUG:boto:Params: {} 2024-04-09T21:52:42.697 DEBUG:boto:Token: None 2024-04-09T21:52:42.697 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33 2024-04-09T21:52:42.697 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc= 2024-04-09T21:52:42.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc=', 'Content-Length': '0'} 2024-04-09T21:52:42.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7c40e1931e68dae-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.699 DEBUG:boto:path=/xryqqk-11/obj33 2024-04-09T21:52:42.699 DEBUG:boto:auth_path=/xryqqk-11/obj33 2024-04-09T21:52:42.699 DEBUG:boto:Method: GET 2024-04-09T21:52:42.699 DEBUG:boto:Path: /xryqqk-11/obj33 2024-04-09T21:52:42.699 DEBUG:boto:Data: 2024-04-09T21:52:42.699 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.699 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.699 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.699 DEBUG:boto:Params: {} 2024-04-09T21:52:42.699 DEBUG:boto:Token: None 2024-04-09T21:52:42.700 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj33 2024-04-09T21:52:42.700 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc= 2024-04-09T21:52:42.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc=', 'Content-Length': '0'} 2024-04-09T21:52:42.701 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab1309b03178998a-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.701 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.701 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.702 DEBUG:boto:path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.702 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.702 DEBUG:boto:Method: GET 2024-04-09T21:52:42.702 DEBUG:boto:Path: /xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.702 DEBUG:boto:Data: 2024-04-09T21:52:42.702 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.702 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.702 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.702 DEBUG:boto:Params: {} 2024-04-09T21:52:42.702 DEBUG:boto:Token: None 2024-04-09T21:52:42.703 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.703 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c95e4beb788d34-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.705 DEBUG:boto:b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.705 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.705 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.705 DEBUG:boto:path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.706 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.706 DEBUG:boto:Method: GET 2024-04-09T21:52:42.706 DEBUG:boto:Path: /xryqqk-11/?prefix=obj33 2024-04-09T21:52:42.706 DEBUG:boto:Data: 2024-04-09T21:52:42.706 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.706 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.706 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.706 DEBUG:boto:Params: {} 2024-04-09T21:52:42.706 DEBUG:boto:Token: None 2024-04-09T21:52:42.706 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.707 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6128743fcc1fa5-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.708 DEBUG:boto:b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-09T21:52:42.709 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.709 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.709 DEBUG:boto:path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.709 DEBUG:boto:auth_path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.709 DEBUG:boto:Method: GET 2024-04-09T21:52:42.709 DEBUG:boto:Path: /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.709 DEBUG:boto:Data: 2024-04-09T21:52:42.709 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.710 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.710 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.710 DEBUG:boto:Params: {} 2024-04-09T21:52:42.710 DEBUG:boto:Token: None 2024-04-09T21:52:42.710 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.710 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-09T21:52:42.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-09T21:52:42.712 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000260169cb358c1ff5-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.712 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.712 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.712 DEBUG:boto:path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.712 DEBUG:boto:auth_path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.712 DEBUG:boto:Method: GET 2024-04-09T21:52:42.712 DEBUG:boto:Path: /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.713 DEBUG:boto:Data: 2024-04-09T21:52:42.713 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.713 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.713 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.713 DEBUG:boto:Params: {} 2024-04-09T21:52:42.713 DEBUG:boto:Token: None 2024-04-09T21:52:42.713 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.714 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-09T21:52:42.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-09T21:52:42.715 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009afad587a7c8bbd3-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.715 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.715 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.716 DEBUG:boto:path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.716 DEBUG:boto:auth_path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.716 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.716 DEBUG:boto:Path: /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.716 DEBUG:boto:Data: 2024-04-09T21:52:42.716 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.716 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.716 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.716 DEBUG:boto:Params: {} 2024-04-09T21:52:42.716 DEBUG:boto:Token: None 2024-04-09T21:52:42.717 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.717 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws= 2024-04-09T21:52:42.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws=', 'Content-Length': '0'} 2024-04-09T21:52:42.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003978c6dfeaaedaff-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.718 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.719 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.719 DEBUG:boto:path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.719 DEBUG:boto:auth_path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.719 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.719 DEBUG:boto:Path: /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.719 DEBUG:boto:Data: 2024-04-09T21:52:42.719 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.719 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.719 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.719 DEBUG:boto:Params: {} 2024-04-09T21:52:42.719 DEBUG:boto:Token: None 2024-04-09T21:52:42.720 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.720 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws= 2024-04-09T21:52:42.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws=', 'Content-Length': '0'} 2024-04-09T21:52:42.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000806d6cff1e6a4dfc-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-09T21:52:42.722 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.722 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.722 DEBUG:boto:path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.722 DEBUG:boto:auth_path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.722 DEBUG:boto:Method: GET 2024-04-09T21:52:42.722 DEBUG:boto:Path: /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.722 DEBUG:boto:Data: 2024-04-09T21:52:42.722 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.723 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.723 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.723 DEBUG:boto:Params: {} 2024-04-09T21:52:42.723 DEBUG:boto:Token: None 2024-04-09T21:52:42.723 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.723 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-09T21:52:42.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-09T21:52:42.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000256097f3bbae3085-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.725 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.725 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.725 DEBUG:boto:path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.725 DEBUG:boto:auth_path=/xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.726 DEBUG:boto:Method: GET 2024-04-09T21:52:42.726 DEBUG:boto:Path: /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.726 DEBUG:boto:Data: 2024-04-09T21:52:42.726 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.726 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.726 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.726 DEBUG:boto:Params: {} 2024-04-09T21:52:42.726 DEBUG:boto:Token: None 2024-04-09T21:52:42.726 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34?versionId=null 2024-04-09T21:52:42.726 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-09T21:52:42.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-09T21:52:42.728 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1c591e7fcc8a899-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.728 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.728 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.729 DEBUG:boto:path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.729 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.729 DEBUG:boto:Method: GET 2024-04-09T21:52:42.729 DEBUG:boto:Path: /xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.729 DEBUG:boto:Data: 2024-04-09T21:52:42.729 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.729 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.729 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.729 DEBUG:boto:Params: {} 2024-04-09T21:52:42.729 DEBUG:boto:Token: None 2024-04-09T21:52:42.729 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.729 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083bcd68fb666572-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.731 DEBUG:boto:b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.732 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.732 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.732 DEBUG:boto:path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.732 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.732 DEBUG:boto:Method: GET 2024-04-09T21:52:42.732 DEBUG:boto:Path: /xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.732 DEBUG:boto:Data: 2024-04-09T21:52:42.733 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.733 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.733 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.733 DEBUG:boto:Params: {} 2024-04-09T21:52:42.733 DEBUG:boto:Token: None 2024-04-09T21:52:42.733 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.733 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5cbac2c45fd5207-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.736 DEBUG:boto:b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.736 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.736 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.736 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.736 DEBUG:boto:Path: /xryqqk-11/obj34 2024-04-09T21:52:42.736 DEBUG:boto:Data: 2024-04-09T21:52:42.736 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.736 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.737 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.737 DEBUG:boto:Params: {} 2024-04-09T21:52:42.737 DEBUG:boto:Token: None 2024-04-09T21:52:42.737 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34 2024-04-09T21:52:42.737 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc= 2024-04-09T21:52:42.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc=', 'Content-Length': '0'} 2024-04-09T21:52:42.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb9cf58b8d09310c-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.739 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.739 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.739 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.739 DEBUG:boto:Path: /xryqqk-11/obj34 2024-04-09T21:52:42.739 DEBUG:boto:Data: 2024-04-09T21:52:42.739 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.740 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.740 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.740 DEBUG:boto:Params: {} 2024-04-09T21:52:42.740 DEBUG:boto:Token: None 2024-04-09T21:52:42.740 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34 2024-04-09T21:52:42.740 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc= 2024-04-09T21:52:42.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc=', 'Content-Length': '0'} 2024-04-09T21:52:42.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef4602b27da1e6aa-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-09T21:52:42.742 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.742 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.742 DEBUG:boto:Method: GET 2024-04-09T21:52:42.742 DEBUG:boto:Path: /xryqqk-11/obj34 2024-04-09T21:52:42.742 DEBUG:boto:Data: 2024-04-09T21:52:42.742 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.742 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.742 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.743 DEBUG:boto:Params: {} 2024-04-09T21:52:42.743 DEBUG:boto:Token: None 2024-04-09T21:52:42.743 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34 2024-04-09T21:52:42.743 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4= 2024-04-09T21:52:42.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4=', 'Content-Length': '0'} 2024-04-09T21:52:42.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076ef3ecace4bc9eb-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.790 DEBUG:boto:path=/xryqqk-11/obj34 2024-04-09T21:52:42.790 DEBUG:boto:auth_path=/xryqqk-11/obj34 2024-04-09T21:52:42.791 DEBUG:boto:Method: GET 2024-04-09T21:52:42.791 DEBUG:boto:Path: /xryqqk-11/obj34 2024-04-09T21:52:42.791 DEBUG:boto:Data: 2024-04-09T21:52:42.791 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.791 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.791 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.791 DEBUG:boto:Params: {} 2024-04-09T21:52:42.791 DEBUG:boto:Token: None 2024-04-09T21:52:42.792 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj34 2024-04-09T21:52:42.792 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4= 2024-04-09T21:52:42.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4=', 'Content-Length': '0'} 2024-04-09T21:52:42.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041c45357796727d9-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.794 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.794 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.794 DEBUG:boto:path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.794 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.794 DEBUG:boto:Method: GET 2024-04-09T21:52:42.794 DEBUG:boto:Path: /xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.794 DEBUG:boto:Data: 2024-04-09T21:52:42.795 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.795 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.795 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.795 DEBUG:boto:Params: {} 2024-04-09T21:52:42.795 DEBUG:boto:Token: None 2024-04-09T21:52:42.795 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.795 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010505f8fd822a202-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.797 DEBUG:boto:b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.798 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.798 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.798 DEBUG:boto:path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.798 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.798 DEBUG:boto:Method: GET 2024-04-09T21:52:42.798 DEBUG:boto:Path: /xryqqk-11/?prefix=obj34 2024-04-09T21:52:42.798 DEBUG:boto:Data: 2024-04-09T21:52:42.798 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.799 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.799 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.799 DEBUG:boto:Params: {} 2024-04-09T21:52:42.800 DEBUG:boto:Token: None 2024-04-09T21:52:42.800 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.800 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed80050929d0fe5-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.802 DEBUG:boto:b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-04-09T21:52:42.803 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.803 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.803 DEBUG:boto:path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.803 DEBUG:boto:auth_path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.803 DEBUG:boto:Method: GET 2024-04-09T21:52:42.803 DEBUG:boto:Path: /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.803 DEBUG:boto:Data: 2024-04-09T21:52:42.803 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.803 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.803 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.803 DEBUG:boto:Params: {} 2024-04-09T21:52:42.804 DEBUG:boto:Token: None 2024-04-09T21:52:42.804 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-09T21:52:42.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-09T21:52:42.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000872d7282d4be376b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.806 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.806 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.806 DEBUG:boto:path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.806 DEBUG:boto:auth_path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.806 DEBUG:boto:Method: GET 2024-04-09T21:52:42.806 DEBUG:boto:Path: /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.806 DEBUG:boto:Data: 2024-04-09T21:52:42.806 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.807 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.807 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.807 DEBUG:boto:Params: {} 2024-04-09T21:52:42.807 DEBUG:boto:Token: None 2024-04-09T21:52:42.807 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.807 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-09T21:52:42.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-09T21:52:42.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d11ae69b35d6a94-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.809 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.810 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.810 DEBUG:boto:path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.810 DEBUG:boto:auth_path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.810 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.810 DEBUG:boto:Path: /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.810 DEBUG:boto:Data: 2024-04-09T21:52:42.810 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.810 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.810 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.810 DEBUG:boto:Params: {} 2024-04-09T21:52:42.811 DEBUG:boto:Token: None 2024-04-09T21:52:42.811 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.811 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug= 2024-04-09T21:52:42.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug=', 'Content-Length': '0'} 2024-04-09T21:52:42.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d71d183811d11553-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.813 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.813 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.813 DEBUG:boto:path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.813 DEBUG:boto:auth_path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.813 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.813 DEBUG:boto:Path: /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.813 DEBUG:boto:Data: 2024-04-09T21:52:42.813 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.813 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.814 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.814 DEBUG:boto:Params: {} 2024-04-09T21:52:42.814 DEBUG:boto:Token: None 2024-04-09T21:52:42.814 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.814 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug= 2024-04-09T21:52:42.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug=', 'Content-Length': '0'} 2024-04-09T21:52:42.816 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d121ba883636ea50-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-04-09T21:52:42.816 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.816 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.816 DEBUG:boto:path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.816 DEBUG:boto:auth_path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.816 DEBUG:boto:Method: GET 2024-04-09T21:52:42.817 DEBUG:boto:Path: /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.817 DEBUG:boto:Data: 2024-04-09T21:52:42.817 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.817 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.817 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.817 DEBUG:boto:Params: {} 2024-04-09T21:52:42.817 DEBUG:boto:Token: None 2024-04-09T21:52:42.817 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.817 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-09T21:52:42.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-09T21:52:42.819 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007080dc45f5f939e0-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.819 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.819 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.819 DEBUG:boto:path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.819 DEBUG:boto:auth_path=/xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.820 DEBUG:boto:Method: GET 2024-04-09T21:52:42.820 DEBUG:boto:Path: /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.820 DEBUG:boto:Data: 2024-04-09T21:52:42.820 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.820 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.820 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.820 DEBUG:boto:Params: {} 2024-04-09T21:52:42.820 DEBUG:boto:Token: None 2024-04-09T21:52:42.820 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35?versionId=null 2024-04-09T21:52:42.820 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-09T21:52:42.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-09T21:52:42.822 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c802ae4d327b39-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.822 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.822 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.822 DEBUG:boto:path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.823 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.823 DEBUG:boto:Method: GET 2024-04-09T21:52:42.823 DEBUG:boto:Path: /xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.823 DEBUG:boto:Data: 2024-04-09T21:52:42.823 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.823 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.823 DEBUG:boto:Params: {} 2024-04-09T21:52:42.823 DEBUG:boto:Token: None 2024-04-09T21:52:42.823 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f016cb4b1483a0-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.826 DEBUG:boto:b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.826 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.826 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.826 DEBUG:boto:path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.826 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.827 DEBUG:boto:Method: GET 2024-04-09T21:52:42.827 DEBUG:boto:Path: /xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.827 DEBUG:boto:Data: 2024-04-09T21:52:42.827 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.827 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.827 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.827 DEBUG:boto:Params: {} 2024-04-09T21:52:42.827 DEBUG:boto:Token: None 2024-04-09T21:52:42.827 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.827 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d94d9c5b87d8b2-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.829 DEBUG:boto:b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.830 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.830 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.830 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.830 DEBUG:boto:Path: /xryqqk-11/obj35 2024-04-09T21:52:42.831 DEBUG:boto:Data: 2024-04-09T21:52:42.831 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.831 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.831 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.831 DEBUG:boto:Params: {} 2024-04-09T21:52:42.831 DEBUG:boto:Token: None 2024-04-09T21:52:42.831 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35 2024-04-09T21:52:42.831 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s= 2024-04-09T21:52:42.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s=', 'Content-Length': '0'} 2024-04-09T21:52:42.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3a91844164f28d7-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.833 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.833 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.833 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.834 DEBUG:boto:Path: /xryqqk-11/obj35 2024-04-09T21:52:42.834 DEBUG:boto:Data: 2024-04-09T21:52:42.834 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.834 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.834 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.834 DEBUG:boto:Params: {} 2024-04-09T21:52:42.834 DEBUG:boto:Token: None 2024-04-09T21:52:42.834 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35 2024-04-09T21:52:42.834 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s= 2024-04-09T21:52:42.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s=', 'Content-Length': '0'} 2024-04-09T21:52:42.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004db24409ae59dd89-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj35 2024-04-09T21:52:42.836 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.836 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.837 DEBUG:boto:Method: GET 2024-04-09T21:52:42.837 DEBUG:boto:Path: /xryqqk-11/obj35 2024-04-09T21:52:42.837 DEBUG:boto:Data: 2024-04-09T21:52:42.837 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.837 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.837 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.837 DEBUG:boto:Params: {} 2024-04-09T21:52:42.837 DEBUG:boto:Token: None 2024-04-09T21:52:42.837 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35 2024-04-09T21:52:42.837 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc= 2024-04-09T21:52:42.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc=', 'Content-Length': '0'} 2024-04-09T21:52:42.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015ee93c0ec92d0ef-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.839 DEBUG:boto:path=/xryqqk-11/obj35 2024-04-09T21:52:42.839 DEBUG:boto:auth_path=/xryqqk-11/obj35 2024-04-09T21:52:42.839 DEBUG:boto:Method: GET 2024-04-09T21:52:42.839 DEBUG:boto:Path: /xryqqk-11/obj35 2024-04-09T21:52:42.840 DEBUG:boto:Data: 2024-04-09T21:52:42.840 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.840 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.840 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.840 DEBUG:boto:Params: {} 2024-04-09T21:52:42.840 DEBUG:boto:Token: None 2024-04-09T21:52:42.840 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj35 2024-04-09T21:52:42.840 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc= 2024-04-09T21:52:42.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc=', 'Content-Length': '0'} 2024-04-09T21:52:42.842 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000031d31a5d6d84f2e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.842 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.842 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.842 DEBUG:boto:path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.843 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.843 DEBUG:boto:Method: GET 2024-04-09T21:52:42.843 DEBUG:boto:Path: /xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.843 DEBUG:boto:Data: 2024-04-09T21:52:42.843 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.843 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.843 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.843 DEBUG:boto:Params: {} 2024-04-09T21:52:42.843 DEBUG:boto:Token: None 2024-04-09T21:52:42.843 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.844 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000706eb23c2535f4e9-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.886 DEBUG:boto:b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.890 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.890 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.890 DEBUG:boto:path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.890 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.890 DEBUG:boto:Method: GET 2024-04-09T21:52:42.891 DEBUG:boto:Path: /xryqqk-11/?prefix=obj35 2024-04-09T21:52:42.891 DEBUG:boto:Data: 2024-04-09T21:52:42.891 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.891 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.891 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.891 DEBUG:boto:Params: {} 2024-04-09T21:52:42.891 DEBUG:boto:Token: None 2024-04-09T21:52:42.891 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.892 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b284f51823dce53d-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.894 DEBUG:boto:b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-04-09T21:52:42.894 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.895 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.895 DEBUG:boto:path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.895 DEBUG:boto:auth_path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.895 DEBUG:boto:Method: GET 2024-04-09T21:52:42.895 DEBUG:boto:Path: /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.895 DEBUG:boto:Data: 2024-04-09T21:52:42.895 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.895 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.895 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.895 DEBUG:boto:Params: {} 2024-04-09T21:52:42.896 DEBUG:boto:Token: None 2024-04-09T21:52:42.896 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.896 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-09T21:52:42.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-09T21:52:42.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa6c72a8e20972db-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.898 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.898 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.898 DEBUG:boto:path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.898 DEBUG:boto:auth_path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.898 DEBUG:boto:Method: GET 2024-04-09T21:52:42.898 DEBUG:boto:Path: /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.898 DEBUG:boto:Data: 2024-04-09T21:52:42.898 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.898 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.899 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.899 DEBUG:boto:Params: {} 2024-04-09T21:52:42.899 DEBUG:boto:Token: None 2024-04-09T21:52:42.899 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.899 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-09T21:52:42.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-09T21:52:42.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000317ec9cf2febf8e6-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.901 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.901 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.901 DEBUG:boto:path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.901 DEBUG:boto:auth_path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.901 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.901 DEBUG:boto:Path: /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.902 DEBUG:boto:Data: 2024-04-09T21:52:42.902 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.902 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.902 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.902 DEBUG:boto:Params: {} 2024-04-09T21:52:42.902 DEBUG:boto:Token: None 2024-04-09T21:52:42.902 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.902 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng= 2024-04-09T21:52:42.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng=', 'Content-Length': '0'} 2024-04-09T21:52:42.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031c77201f2feb0ef-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.904 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.904 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.904 DEBUG:boto:path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.904 DEBUG:boto:auth_path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.904 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.905 DEBUG:boto:Path: /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.905 DEBUG:boto:Data: 2024-04-09T21:52:42.905 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.905 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.905 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.905 DEBUG:boto:Params: {} 2024-04-09T21:52:42.905 DEBUG:boto:Token: None 2024-04-09T21:52:42.905 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.905 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng= 2024-04-09T21:52:42.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng=', 'Content-Length': '0'} 2024-04-09T21:52:42.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd4144fd1b2f7fb0-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-04-09T21:52:42.907 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.907 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.907 DEBUG:boto:path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.907 DEBUG:boto:auth_path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.908 DEBUG:boto:Method: GET 2024-04-09T21:52:42.908 DEBUG:boto:Path: /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.908 DEBUG:boto:Data: 2024-04-09T21:52:42.908 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.908 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.908 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.908 DEBUG:boto:Params: {} 2024-04-09T21:52:42.908 DEBUG:boto:Token: None 2024-04-09T21:52:42.908 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.908 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-09T21:52:42.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-09T21:52:42.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019d4ce635d20ef68-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.911 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.911 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.911 DEBUG:boto:path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.911 DEBUG:boto:auth_path=/xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.911 DEBUG:boto:Method: GET 2024-04-09T21:52:42.911 DEBUG:boto:Path: /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.911 DEBUG:boto:Data: 2024-04-09T21:52:42.912 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.912 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.912 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.912 DEBUG:boto:Params: {} 2024-04-09T21:52:42.912 DEBUG:boto:Token: None 2024-04-09T21:52:42.912 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36?versionId=null 2024-04-09T21:52:42.912 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-09T21:52:42.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-09T21:52:42.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c80af881f059b942-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.914 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.915 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.915 DEBUG:boto:path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.915 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.915 DEBUG:boto:Method: GET 2024-04-09T21:52:42.915 DEBUG:boto:Path: /xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.915 DEBUG:boto:Data: 2024-04-09T21:52:42.915 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.915 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.915 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.915 DEBUG:boto:Params: {} 2024-04-09T21:52:42.916 DEBUG:boto:Token: None 2024-04-09T21:52:42.916 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.916 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d616a5cb7b8c08-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.918 DEBUG:boto:b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.919 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.919 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.919 DEBUG:boto:path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.919 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.919 DEBUG:boto:Method: GET 2024-04-09T21:52:42.919 DEBUG:boto:Path: /xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.919 DEBUG:boto:Data: 2024-04-09T21:52:42.919 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.919 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.919 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.919 DEBUG:boto:Params: {} 2024-04-09T21:52:42.920 DEBUG:boto:Token: None 2024-04-09T21:52:42.920 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.920 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000051b111d53951044-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.922 DEBUG:boto:b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.923 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.923 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.923 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.923 DEBUG:boto:Path: /xryqqk-11/obj36 2024-04-09T21:52:42.923 DEBUG:boto:Data: 2024-04-09T21:52:42.923 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.923 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.923 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.923 DEBUG:boto:Params: {} 2024-04-09T21:52:42.923 DEBUG:boto:Token: None 2024-04-09T21:52:42.924 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36 2024-04-09T21:52:42.924 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA= 2024-04-09T21:52:42.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA=', 'Content-Length': '0'} 2024-04-09T21:52:42.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055b0ee13cefa6c46-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.926 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.926 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.926 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.926 DEBUG:boto:Path: /xryqqk-11/obj36 2024-04-09T21:52:42.926 DEBUG:boto:Data: 2024-04-09T21:52:42.926 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.926 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.926 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.926 DEBUG:boto:Params: {} 2024-04-09T21:52:42.927 DEBUG:boto:Token: None 2024-04-09T21:52:42.927 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36 2024-04-09T21:52:42.927 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA= 2024-04-09T21:52:42.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA=', 'Content-Length': '0'} 2024-04-09T21:52:42.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004009840f6155219d-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj36 2024-04-09T21:52:42.929 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.929 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.929 DEBUG:boto:Method: GET 2024-04-09T21:52:42.929 DEBUG:boto:Path: /xryqqk-11/obj36 2024-04-09T21:52:42.929 DEBUG:boto:Data: 2024-04-09T21:52:42.929 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.929 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.929 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.930 DEBUG:boto:Params: {} 2024-04-09T21:52:42.930 DEBUG:boto:Token: None 2024-04-09T21:52:42.930 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36 2024-04-09T21:52:42.930 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM= 2024-04-09T21:52:42.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM=', 'Content-Length': '0'} 2024-04-09T21:52:42.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045d9e1e52f94c970-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.932 DEBUG:boto:path=/xryqqk-11/obj36 2024-04-09T21:52:42.932 DEBUG:boto:auth_path=/xryqqk-11/obj36 2024-04-09T21:52:42.933 DEBUG:boto:Method: GET 2024-04-09T21:52:42.933 DEBUG:boto:Path: /xryqqk-11/obj36 2024-04-09T21:52:42.933 DEBUG:boto:Data: 2024-04-09T21:52:42.933 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.933 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.933 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.933 DEBUG:boto:Params: {} 2024-04-09T21:52:42.933 DEBUG:boto:Token: None 2024-04-09T21:52:42.933 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj36 2024-04-09T21:52:42.934 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM= 2024-04-09T21:52:42.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM=', 'Content-Length': '0'} 2024-04-09T21:52:42.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094ed55767f58bc0e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.935 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.936 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.936 DEBUG:boto:path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.936 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.936 DEBUG:boto:Method: GET 2024-04-09T21:52:42.936 DEBUG:boto:Path: /xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.936 DEBUG:boto:Data: 2024-04-09T21:52:42.936 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.936 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.936 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.936 DEBUG:boto:Params: {} 2024-04-09T21:52:42.937 DEBUG:boto:Token: None 2024-04-09T21:52:42.937 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.937 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff248fb2f61dfa96-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.939 DEBUG:boto:b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.939 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:42.940 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:42.940 DEBUG:boto:path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.940 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.940 DEBUG:boto:Method: GET 2024-04-09T21:52:42.940 DEBUG:boto:Path: /xryqqk-11/?prefix=obj36 2024-04-09T21:52:42.940 DEBUG:boto:Data: 2024-04-09T21:52:42.940 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.940 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.940 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.940 DEBUG:boto:Params: {} 2024-04-09T21:52:42.941 DEBUG:boto:Token: None 2024-04-09T21:52:42.941 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/ 2024-04-09T21:52:42.941 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-09T21:52:42.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-09T21:52:42.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c614a57dd3515b-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.943 DEBUG:boto:b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:42.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-09T21:52:42.943 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:42.944 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:42.944 DEBUG:boto:path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.944 DEBUG:boto:auth_path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.944 DEBUG:boto:Method: GET 2024-04-09T21:52:42.944 DEBUG:boto:Path: /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.944 DEBUG:boto:Data: 2024-04-09T21:52:42.944 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.944 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.944 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.944 DEBUG:boto:Params: {} 2024-04-09T21:52:42.945 DEBUG:boto:Token: None 2024-04-09T21:52:42.945 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.945 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI= 2024-04-09T21:52:42.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI=', 'Content-Length': '0'} 2024-04-09T21:52:42.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000402c8d30d921e9f0-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.990 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:42.991 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:42.991 DEBUG:boto:path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.991 DEBUG:boto:auth_path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.991 DEBUG:boto:Method: GET 2024-04-09T21:52:42.991 DEBUG:boto:Path: /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.991 DEBUG:boto:Data: 2024-04-09T21:52:42.991 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.991 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.991 DEBUG:boto:Port: 8000 2024-04-09T21:52:42.991 DEBUG:boto:Params: {} 2024-04-09T21:52:42.991 DEBUG:boto:Token: None 2024-04-09T21:52:42.992 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.992 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI= 2024-04-09T21:52:42.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI=', 'Content-Length': '0'} 2024-04-09T21:52:42.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009679fca4eb8ddc96-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.995 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:42.995 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:42.995 DEBUG:boto:path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.995 DEBUG:boto:auth_path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.995 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.995 DEBUG:boto:Path: /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.995 DEBUG:boto:Data: 2024-04-09T21:52:42.996 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.996 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:42.996 DEBUG:boto:Port: 8001 2024-04-09T21:52:42.996 DEBUG:boto:Params: {} 2024-04-09T21:52:42.996 DEBUG:boto:Token: None 2024-04-09T21:52:42.996 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:42 GMT /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.996 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y7X1tfe811ftdYY8Eyyb5GThhjM= 2024-04-09T21:52:42.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y7X1tfe811ftdYY8Eyyb5GThhjM=', 'Content-Length': '0'} 2024-04-09T21:52:42.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fc56e47d68273b4-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:42.998 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:42.999 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:42.999 DEBUG:boto:path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.999 DEBUG:boto:auth_path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.999 DEBUG:boto:Method: HEAD 2024-04-09T21:52:42.999 DEBUG:boto:Path: /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:42.999 DEBUG:boto:Data: 2024-04-09T21:52:42.999 DEBUG:boto:Headers: {} 2024-04-09T21:52:42.999 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:42.999 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.000 DEBUG:boto:Params: {} 2024-04-09T21:52:43.000 DEBUG:boto:Token: None 2024-04-09T21:52:43.000 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.000 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aZyfdRdPmHP19bXLpGDiCSh+Fzg= 2024-04-09T21:52:43.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aZyfdRdPmHP19bXLpGDiCSh+Fzg=', 'Content-Length': '0'} 2024-04-09T21:52:43.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6478ba9ed29b903-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-09T21:52:43.002 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:43.003 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:43.003 DEBUG:boto:path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.003 DEBUG:boto:auth_path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.003 DEBUG:boto:Method: GET 2024-04-09T21:52:43.003 DEBUG:boto:Path: /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.003 DEBUG:boto:Data: 2024-04-09T21:52:43.003 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.003 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.003 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.003 DEBUG:boto:Params: {} 2024-04-09T21:52:43.003 DEBUG:boto:Token: None 2024-04-09T21:52:43.004 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.004 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8= 2024-04-09T21:52:43.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8=', 'Content-Length': '0'} 2024-04-09T21:52:43.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cc4f272f9014a2d-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.006 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:43.006 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:43.006 DEBUG:boto:path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.006 DEBUG:boto:auth_path=/xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.006 DEBUG:boto:Method: GET 2024-04-09T21:52:43.006 DEBUG:boto:Path: /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.006 DEBUG:boto:Data: 2024-04-09T21:52:43.006 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.006 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.006 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.006 DEBUG:boto:Params: {} 2024-04-09T21:52:43.007 DEBUG:boto:Token: None 2024-04-09T21:52:43.007 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37?versionId=null 2024-04-09T21:52:43.007 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8= 2024-04-09T21:52:43.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8=', 'Content-Length': '0'} 2024-04-09T21:52:43.010 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040b4353d6b56e398-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.011 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.011 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.011 DEBUG:boto:path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.011 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.011 DEBUG:boto:Method: GET 2024-04-09T21:52:43.011 DEBUG:boto:Path: /xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.012 DEBUG:boto:Data: 2024-04-09T21:52:43.012 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.012 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.012 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.012 DEBUG:boto:Params: {} 2024-04-09T21:52:43.012 DEBUG:boto:Token: None 2024-04-09T21:52:43.012 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.013 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092285209e1a5365a-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.018 DEBUG:boto:b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.019 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.019 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.019 DEBUG:boto:path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.019 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.019 DEBUG:boto:Method: GET 2024-04-09T21:52:43.020 DEBUG:boto:Path: /xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.020 DEBUG:boto:Data: 2024-04-09T21:52:43.020 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.020 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.020 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.020 DEBUG:boto:Params: {} 2024-04-09T21:52:43.020 DEBUG:boto:Token: None 2024-04-09T21:52:43.020 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.020 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270ef9dc7430ba99-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.023 DEBUG:boto:b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.023 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:43.023 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:43.023 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.023 DEBUG:boto:Path: /xryqqk-11/obj37 2024-04-09T21:52:43.023 DEBUG:boto:Data: 2024-04-09T21:52:43.023 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.024 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.024 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.024 DEBUG:boto:Params: {} 2024-04-09T21:52:43.024 DEBUG:boto:Token: None 2024-04-09T21:52:43.024 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37 2024-04-09T21:52:43.024 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0= 2024-04-09T21:52:43.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0=', 'Content-Length': '0'} 2024-04-09T21:52:43.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce0fff2fc0387615-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.026 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:43.026 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:43.026 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.026 DEBUG:boto:Path: /xryqqk-11/obj37 2024-04-09T21:52:43.026 DEBUG:boto:Data: 2024-04-09T21:52:43.026 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.026 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.027 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.027 DEBUG:boto:Params: {} 2024-04-09T21:52:43.027 DEBUG:boto:Token: None 2024-04-09T21:52:43.027 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37 2024-04-09T21:52:43.027 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0= 2024-04-09T21:52:43.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0=', 'Content-Length': '0'} 2024-04-09T21:52:43.028 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000745ce149aa237b58-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-09T21:52:43.029 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:43.029 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:43.029 DEBUG:boto:Method: GET 2024-04-09T21:52:43.029 DEBUG:boto:Path: /xryqqk-11/obj37 2024-04-09T21:52:43.029 DEBUG:boto:Data: 2024-04-09T21:52:43.029 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.029 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.029 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.029 DEBUG:boto:Params: {} 2024-04-09T21:52:43.029 DEBUG:boto:Token: None 2024-04-09T21:52:43.030 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37 2024-04-09T21:52:43.030 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o= 2024-04-09T21:52:43.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o=', 'Content-Length': '0'} 2024-04-09T21:52:43.031 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a84c1dab9938791-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.032 DEBUG:boto:path=/xryqqk-11/obj37 2024-04-09T21:52:43.032 DEBUG:boto:auth_path=/xryqqk-11/obj37 2024-04-09T21:52:43.032 DEBUG:boto:Method: GET 2024-04-09T21:52:43.032 DEBUG:boto:Path: /xryqqk-11/obj37 2024-04-09T21:52:43.032 DEBUG:boto:Data: 2024-04-09T21:52:43.032 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.032 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.032 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.032 DEBUG:boto:Params: {} 2024-04-09T21:52:43.032 DEBUG:boto:Token: None 2024-04-09T21:52:43.033 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj37 2024-04-09T21:52:43.033 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o= 2024-04-09T21:52:43.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o=', 'Content-Length': '0'} 2024-04-09T21:52:43.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e3eb6a7f59a7de3-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.035 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.035 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.035 DEBUG:boto:path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.035 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.035 DEBUG:boto:Method: GET 2024-04-09T21:52:43.035 DEBUG:boto:Path: /xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.035 DEBUG:boto:Data: 2024-04-09T21:52:43.035 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.035 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.036 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.036 DEBUG:boto:Params: {} 2024-04-09T21:52:43.036 DEBUG:boto:Token: None 2024-04-09T21:52:43.036 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.036 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d807bec66b70e4-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.038 DEBUG:boto:b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.041 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.041 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.041 DEBUG:boto:path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.041 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.041 DEBUG:boto:Method: GET 2024-04-09T21:52:43.041 DEBUG:boto:Path: /xryqqk-11/?prefix=obj37 2024-04-09T21:52:43.042 DEBUG:boto:Data: 2024-04-09T21:52:43.042 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.042 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.042 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.042 DEBUG:boto:Params: {} 2024-04-09T21:52:43.042 DEBUG:boto:Token: None 2024-04-09T21:52:43.042 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.042 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1bb8b19b889103-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.086 DEBUG:boto:b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-09T21:52:43.087 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.087 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.087 DEBUG:boto:path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.087 DEBUG:boto:auth_path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.087 DEBUG:boto:Method: GET 2024-04-09T21:52:43.087 DEBUG:boto:Path: /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.088 DEBUG:boto:Data: 2024-04-09T21:52:43.088 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.088 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.088 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.088 DEBUG:boto:Params: {} 2024-04-09T21:52:43.088 DEBUG:boto:Token: None 2024-04-09T21:52:43.088 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.088 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-09T21:52:43.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-09T21:52:43.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e80653824eaa3c6-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.090 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.091 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.091 DEBUG:boto:path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.091 DEBUG:boto:auth_path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.091 DEBUG:boto:Method: GET 2024-04-09T21:52:43.091 DEBUG:boto:Path: /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.091 DEBUG:boto:Data: 2024-04-09T21:52:43.091 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.091 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.091 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.092 DEBUG:boto:Params: {} 2024-04-09T21:52:43.092 DEBUG:boto:Token: None 2024-04-09T21:52:43.092 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.092 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-09T21:52:43.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-09T21:52:43.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b86f6d5c554a2cf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.094 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.094 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.094 DEBUG:boto:path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.094 DEBUG:boto:auth_path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.094 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.095 DEBUG:boto:Path: /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.095 DEBUG:boto:Data: 2024-04-09T21:52:43.095 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.095 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.095 DEBUG:boto:Params: {} 2024-04-09T21:52:43.095 DEBUG:boto:Token: None 2024-04-09T21:52:43.095 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.095 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws= 2024-04-09T21:52:43.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws=', 'Content-Length': '0'} 2024-04-09T21:52:43.097 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000109190efc73f0805-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.097 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.097 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.097 DEBUG:boto:path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.097 DEBUG:boto:auth_path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.097 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.098 DEBUG:boto:Path: /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.098 DEBUG:boto:Data: 2024-04-09T21:52:43.098 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.098 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.098 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.098 DEBUG:boto:Params: {} 2024-04-09T21:52:43.098 DEBUG:boto:Token: None 2024-04-09T21:52:43.098 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.098 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws= 2024-04-09T21:52:43.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws=', 'Content-Length': '0'} 2024-04-09T21:52:43.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b69ed1678f1346ed-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-09T21:52:43.101 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.101 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.101 DEBUG:boto:path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.101 DEBUG:boto:auth_path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.101 DEBUG:boto:Method: GET 2024-04-09T21:52:43.101 DEBUG:boto:Path: /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.101 DEBUG:boto:Data: 2024-04-09T21:52:43.101 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.101 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.101 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.102 DEBUG:boto:Params: {} 2024-04-09T21:52:43.102 DEBUG:boto:Token: None 2024-04-09T21:52:43.102 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.102 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-09T21:52:43.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-09T21:52:43.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097c2e63c9a29ea32-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.146 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.147 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.147 DEBUG:boto:path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.147 DEBUG:boto:auth_path=/xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.147 DEBUG:boto:Method: GET 2024-04-09T21:52:43.147 DEBUG:boto:Path: /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.147 DEBUG:boto:Data: 2024-04-09T21:52:43.147 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.147 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.147 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.147 DEBUG:boto:Params: {} 2024-04-09T21:52:43.148 DEBUG:boto:Token: None 2024-04-09T21:52:43.148 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38?versionId=null 2024-04-09T21:52:43.148 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-09T21:52:43.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-09T21:52:43.150 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b01dadcc9886a473-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.150 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.150 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.150 DEBUG:boto:path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.150 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.151 DEBUG:boto:Method: GET 2024-04-09T21:52:43.151 DEBUG:boto:Path: /xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.151 DEBUG:boto:Data: 2024-04-09T21:52:43.151 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.151 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.151 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.151 DEBUG:boto:Params: {} 2024-04-09T21:52:43.151 DEBUG:boto:Token: None 2024-04-09T21:52:43.151 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.151 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000819f3db4749d4287-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.154 DEBUG:boto:b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.154 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.154 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.154 DEBUG:boto:path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.154 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.154 DEBUG:boto:Method: GET 2024-04-09T21:52:43.155 DEBUG:boto:Path: /xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.155 DEBUG:boto:Data: 2024-04-09T21:52:43.155 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.155 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.155 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.155 DEBUG:boto:Params: {} 2024-04-09T21:52:43.155 DEBUG:boto:Token: None 2024-04-09T21:52:43.155 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.155 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911baa059a025679-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.157 DEBUG:boto:b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.158 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.158 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.158 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.158 DEBUG:boto:Path: /xryqqk-11/obj38 2024-04-09T21:52:43.158 DEBUG:boto:Data: 2024-04-09T21:52:43.158 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.158 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.158 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.159 DEBUG:boto:Params: {} 2024-04-09T21:52:43.159 DEBUG:boto:Token: None 2024-04-09T21:52:43.159 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38 2024-04-09T21:52:43.159 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc= 2024-04-09T21:52:43.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc=', 'Content-Length': '0'} 2024-04-09T21:52:43.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f38e79c84fa6730-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.161 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.161 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.161 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.161 DEBUG:boto:Path: /xryqqk-11/obj38 2024-04-09T21:52:43.161 DEBUG:boto:Data: 2024-04-09T21:52:43.161 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.161 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.161 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.162 DEBUG:boto:Params: {} 2024-04-09T21:52:43.162 DEBUG:boto:Token: None 2024-04-09T21:52:43.162 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38 2024-04-09T21:52:43.162 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc= 2024-04-09T21:52:43.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc=', 'Content-Length': '0'} 2024-04-09T21:52:43.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd96acc4e81f004-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-09T21:52:43.164 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.164 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.164 DEBUG:boto:Method: GET 2024-04-09T21:52:43.164 DEBUG:boto:Path: /xryqqk-11/obj38 2024-04-09T21:52:43.164 DEBUG:boto:Data: 2024-04-09T21:52:43.164 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.165 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.165 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.165 DEBUG:boto:Params: {} 2024-04-09T21:52:43.165 DEBUG:boto:Token: None 2024-04-09T21:52:43.165 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38 2024-04-09T21:52:43.165 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g= 2024-04-09T21:52:43.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g=', 'Content-Length': '0'} 2024-04-09T21:52:43.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d210b15343a48c7-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.167 DEBUG:boto:path=/xryqqk-11/obj38 2024-04-09T21:52:43.167 DEBUG:boto:auth_path=/xryqqk-11/obj38 2024-04-09T21:52:43.167 DEBUG:boto:Method: GET 2024-04-09T21:52:43.167 DEBUG:boto:Path: /xryqqk-11/obj38 2024-04-09T21:52:43.167 DEBUG:boto:Data: 2024-04-09T21:52:43.167 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.168 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.168 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.168 DEBUG:boto:Params: {} 2024-04-09T21:52:43.168 DEBUG:boto:Token: None 2024-04-09T21:52:43.168 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj38 2024-04-09T21:52:43.168 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g= 2024-04-09T21:52:43.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g=', 'Content-Length': '0'} 2024-04-09T21:52:43.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055aa96410e341041-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.170 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.170 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.170 DEBUG:boto:path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.170 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.170 DEBUG:boto:Method: GET 2024-04-09T21:52:43.171 DEBUG:boto:Path: /xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.171 DEBUG:boto:Data: 2024-04-09T21:52:43.171 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.171 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.171 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.171 DEBUG:boto:Params: {} 2024-04-09T21:52:43.171 DEBUG:boto:Token: None 2024-04-09T21:52:43.171 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.171 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a40c63c1c520d0-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.173 DEBUG:boto:b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.174 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.174 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.174 DEBUG:boto:path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.174 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.174 DEBUG:boto:Method: GET 2024-04-09T21:52:43.174 DEBUG:boto:Path: /xryqqk-11/?prefix=obj38 2024-04-09T21:52:43.175 DEBUG:boto:Data: 2024-04-09T21:52:43.175 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.175 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.175 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.175 DEBUG:boto:Params: {} 2024-04-09T21:52:43.175 DEBUG:boto:Token: None 2024-04-09T21:52:43.175 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.175 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e90b496223dfa7a-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.177 DEBUG:boto:b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-04-09T21:52:43.178 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.178 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.178 DEBUG:boto:path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.178 DEBUG:boto:auth_path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.178 DEBUG:boto:Method: GET 2024-04-09T21:52:43.178 DEBUG:boto:Path: /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.179 DEBUG:boto:Data: 2024-04-09T21:52:43.179 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.179 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.179 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.179 DEBUG:boto:Params: {} 2024-04-09T21:52:43.179 DEBUG:boto:Token: None 2024-04-09T21:52:43.179 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.179 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-09T21:52:43.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-09T21:52:43.181 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013ce626d5606d21e-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.181 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.182 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.182 DEBUG:boto:path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.182 DEBUG:boto:auth_path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.182 DEBUG:boto:Method: GET 2024-04-09T21:52:43.182 DEBUG:boto:Path: /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.182 DEBUG:boto:Data: 2024-04-09T21:52:43.182 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.182 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.182 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.182 DEBUG:boto:Params: {} 2024-04-09T21:52:43.183 DEBUG:boto:Token: None 2024-04-09T21:52:43.183 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.183 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-09T21:52:43.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-09T21:52:43.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076f2eaa2b04afe5f-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.184 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.185 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.185 DEBUG:boto:path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.185 DEBUG:boto:auth_path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.185 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.185 DEBUG:boto:Path: /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.185 DEBUG:boto:Data: 2024-04-09T21:52:43.185 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.186 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.186 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.186 DEBUG:boto:Params: {} 2024-04-09T21:52:43.186 DEBUG:boto:Token: None 2024-04-09T21:52:43.186 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.187 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU= 2024-04-09T21:52:43.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU=', 'Content-Length': '0'} 2024-04-09T21:52:43.188 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002865f6a3daafa27e-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.189 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.189 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.189 DEBUG:boto:path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.189 DEBUG:boto:auth_path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.189 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.189 DEBUG:boto:Path: /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.189 DEBUG:boto:Data: 2024-04-09T21:52:43.189 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.189 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.189 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.189 DEBUG:boto:Params: {} 2024-04-09T21:52:43.190 DEBUG:boto:Token: None 2024-04-09T21:52:43.190 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.190 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU= 2024-04-09T21:52:43.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU=', 'Content-Length': '0'} 2024-04-09T21:52:43.192 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b319e0fa839dfccf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-04-09T21:52:43.192 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.192 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.192 DEBUG:boto:path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.192 DEBUG:boto:auth_path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.192 DEBUG:boto:Method: GET 2024-04-09T21:52:43.192 DEBUG:boto:Path: /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.192 DEBUG:boto:Data: 2024-04-09T21:52:43.193 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.193 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.193 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.193 DEBUG:boto:Params: {} 2024-04-09T21:52:43.193 DEBUG:boto:Token: None 2024-04-09T21:52:43.193 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.193 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-09T21:52:43.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-09T21:52:43.195 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cd996858eee2f1b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.195 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.195 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.196 DEBUG:boto:path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.196 DEBUG:boto:auth_path=/xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.196 DEBUG:boto:Method: GET 2024-04-09T21:52:43.196 DEBUG:boto:Path: /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.196 DEBUG:boto:Data: 2024-04-09T21:52:43.196 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.196 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.196 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.196 DEBUG:boto:Params: {} 2024-04-09T21:52:43.196 DEBUG:boto:Token: None 2024-04-09T21:52:43.197 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39?versionId=null 2024-04-09T21:52:43.197 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-09T21:52:43.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-09T21:52:43.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013d9919892657597-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.199 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.199 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.199 DEBUG:boto:path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.199 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.199 DEBUG:boto:Method: GET 2024-04-09T21:52:43.199 DEBUG:boto:Path: /xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.199 DEBUG:boto:Data: 2024-04-09T21:52:43.199 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.199 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.200 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.200 DEBUG:boto:Params: {} 2024-04-09T21:52:43.200 DEBUG:boto:Token: None 2024-04-09T21:52:43.200 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.200 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2ddd0fd161c94a-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.250 DEBUG:boto:b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.251 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.251 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.251 DEBUG:boto:path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.252 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.252 DEBUG:boto:Method: GET 2024-04-09T21:52:43.252 DEBUG:boto:Path: /xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.252 DEBUG:boto:Data: 2024-04-09T21:52:43.252 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.252 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.252 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.252 DEBUG:boto:Params: {} 2024-04-09T21:52:43.252 DEBUG:boto:Token: None 2024-04-09T21:52:43.252 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.253 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d09c93176c8af1c3-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.255 DEBUG:boto:b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.255 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.256 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.256 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.256 DEBUG:boto:Path: /xryqqk-11/obj39 2024-04-09T21:52:43.256 DEBUG:boto:Data: 2024-04-09T21:52:43.256 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.256 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.256 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.256 DEBUG:boto:Params: {} 2024-04-09T21:52:43.257 DEBUG:boto:Token: None 2024-04-09T21:52:43.257 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39 2024-04-09T21:52:43.257 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA= 2024-04-09T21:52:43.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA=', 'Content-Length': '0'} 2024-04-09T21:52:43.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001250bce39bff58b2-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.259 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.259 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.259 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.259 DEBUG:boto:Path: /xryqqk-11/obj39 2024-04-09T21:52:43.259 DEBUG:boto:Data: 2024-04-09T21:52:43.259 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.259 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.259 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.260 DEBUG:boto:Params: {} 2024-04-09T21:52:43.260 DEBUG:boto:Token: None 2024-04-09T21:52:43.260 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39 2024-04-09T21:52:43.260 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA= 2024-04-09T21:52:43.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA=', 'Content-Length': '0'} 2024-04-09T21:52:43.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa21be20f20e943d-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj39 2024-04-09T21:52:43.262 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.262 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.262 DEBUG:boto:Method: GET 2024-04-09T21:52:43.262 DEBUG:boto:Path: /xryqqk-11/obj39 2024-04-09T21:52:43.262 DEBUG:boto:Data: 2024-04-09T21:52:43.262 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.262 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.262 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.263 DEBUG:boto:Params: {} 2024-04-09T21:52:43.263 DEBUG:boto:Token: None 2024-04-09T21:52:43.263 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39 2024-04-09T21:52:43.263 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ= 2024-04-09T21:52:43.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ=', 'Content-Length': '0'} 2024-04-09T21:52:43.265 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076371154d9cb4312-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.265 DEBUG:boto:path=/xryqqk-11/obj39 2024-04-09T21:52:43.265 DEBUG:boto:auth_path=/xryqqk-11/obj39 2024-04-09T21:52:43.265 DEBUG:boto:Method: GET 2024-04-09T21:52:43.266 DEBUG:boto:Path: /xryqqk-11/obj39 2024-04-09T21:52:43.266 DEBUG:boto:Data: 2024-04-09T21:52:43.266 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.266 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.266 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.266 DEBUG:boto:Params: {} 2024-04-09T21:52:43.267 DEBUG:boto:Token: None 2024-04-09T21:52:43.267 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj39 2024-04-09T21:52:43.267 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ= 2024-04-09T21:52:43.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ=', 'Content-Length': '0'} 2024-04-09T21:52:43.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c61f90e4ec486ce8-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.269 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.269 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.270 DEBUG:boto:path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.270 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.270 DEBUG:boto:Method: GET 2024-04-09T21:52:43.270 DEBUG:boto:Path: /xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.270 DEBUG:boto:Data: 2024-04-09T21:52:43.270 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.270 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.270 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.270 DEBUG:boto:Params: {} 2024-04-09T21:52:43.271 DEBUG:boto:Token: None 2024-04-09T21:52:43.271 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.271 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1288f2a5736fd4-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.273 DEBUG:boto:b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.274 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.274 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.274 DEBUG:boto:path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.274 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.274 DEBUG:boto:Method: GET 2024-04-09T21:52:43.274 DEBUG:boto:Path: /xryqqk-11/?prefix=obj39 2024-04-09T21:52:43.275 DEBUG:boto:Data: 2024-04-09T21:52:43.275 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.275 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.275 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.275 DEBUG:boto:Params: {} 2024-04-09T21:52:43.275 DEBUG:boto:Token: None 2024-04-09T21:52:43.275 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.275 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015ff77c296217def-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.278 DEBUG:boto:b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-09T21:52:43.278 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.278 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.278 DEBUG:boto:path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.279 DEBUG:boto:auth_path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.279 DEBUG:boto:Method: GET 2024-04-09T21:52:43.279 DEBUG:boto:Path: /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.279 DEBUG:boto:Data: 2024-04-09T21:52:43.279 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.279 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.279 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.279 DEBUG:boto:Params: {} 2024-04-09T21:52:43.279 DEBUG:boto:Token: None 2024-04-09T21:52:43.279 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.280 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-09T21:52:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-09T21:52:43.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005067dd8464622edf-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.281 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.282 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.282 DEBUG:boto:path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.282 DEBUG:boto:auth_path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.282 DEBUG:boto:Method: GET 2024-04-09T21:52:43.282 DEBUG:boto:Path: /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.282 DEBUG:boto:Data: 2024-04-09T21:52:43.282 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.282 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.282 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.282 DEBUG:boto:Params: {} 2024-04-09T21:52:43.283 DEBUG:boto:Token: None 2024-04-09T21:52:43.283 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.283 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-09T21:52:43.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-09T21:52:43.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e275bfc3b73e19a-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.285 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.285 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.285 DEBUG:boto:path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.285 DEBUG:boto:auth_path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.285 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.286 DEBUG:boto:Path: /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.286 DEBUG:boto:Data: 2024-04-09T21:52:43.286 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.286 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.286 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.286 DEBUG:boto:Params: {} 2024-04-09T21:52:43.286 DEBUG:boto:Token: None 2024-04-09T21:52:43.286 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.286 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I= 2024-04-09T21:52:43.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I=', 'Content-Length': '0'} 2024-04-09T21:52:43.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2dfa7f8279d3a5a-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.288 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.288 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.289 DEBUG:boto:path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.289 DEBUG:boto:auth_path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.289 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.289 DEBUG:boto:Path: /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.289 DEBUG:boto:Data: 2024-04-09T21:52:43.289 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.289 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.289 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.289 DEBUG:boto:Params: {} 2024-04-09T21:52:43.289 DEBUG:boto:Token: None 2024-04-09T21:52:43.290 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.290 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I= 2024-04-09T21:52:43.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I=', 'Content-Length': '0'} 2024-04-09T21:52:43.291 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038e8b0fa85d41594-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-09T21:52:43.292 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.292 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.292 DEBUG:boto:path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.292 DEBUG:boto:auth_path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.292 DEBUG:boto:Method: GET 2024-04-09T21:52:43.292 DEBUG:boto:Path: /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.293 DEBUG:boto:Data: 2024-04-09T21:52:43.293 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.293 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.293 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.293 DEBUG:boto:Params: {} 2024-04-09T21:52:43.293 DEBUG:boto:Token: None 2024-04-09T21:52:43.293 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.293 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-09T21:52:43.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-09T21:52:43.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a280f83cf851ac82-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.295 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.295 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.296 DEBUG:boto:path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.296 DEBUG:boto:auth_path=/xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.296 DEBUG:boto:Method: GET 2024-04-09T21:52:43.296 DEBUG:boto:Path: /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.296 DEBUG:boto:Data: 2024-04-09T21:52:43.296 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.296 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.297 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.297 DEBUG:boto:Params: {} 2024-04-09T21:52:43.297 DEBUG:boto:Token: None 2024-04-09T21:52:43.297 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4?versionId=null 2024-04-09T21:52:43.297 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-09T21:52:43.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-09T21:52:43.299 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fb948901bb15ddf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.302 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.302 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.302 DEBUG:boto:path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.302 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.302 DEBUG:boto:Method: GET 2024-04-09T21:52:43.302 DEBUG:boto:Path: /xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.302 DEBUG:boto:Data: 2024-04-09T21:52:43.302 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.303 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.303 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.303 DEBUG:boto:Params: {} 2024-04-09T21:52:43.303 DEBUG:boto:Token: None 2024-04-09T21:52:43.303 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.303 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001215a650ce5dda9d-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.306 DEBUG:boto:b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.306 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.307 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.307 DEBUG:boto:path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.307 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.307 DEBUG:boto:Method: GET 2024-04-09T21:52:43.307 DEBUG:boto:Path: /xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.307 DEBUG:boto:Data: 2024-04-09T21:52:43.307 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.307 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.307 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.307 DEBUG:boto:Params: {} 2024-04-09T21:52:43.307 DEBUG:boto:Token: None 2024-04-09T21:52:43.308 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.308 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce32dd4921f3b618-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.310 DEBUG:boto:b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.310 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.311 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.311 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.311 DEBUG:boto:Path: /xryqqk-11/obj4 2024-04-09T21:52:43.311 DEBUG:boto:Data: 2024-04-09T21:52:43.311 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.311 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.311 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.311 DEBUG:boto:Params: {} 2024-04-09T21:52:43.311 DEBUG:boto:Token: None 2024-04-09T21:52:43.311 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4 2024-04-09T21:52:43.312 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE= 2024-04-09T21:52:43.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE=', 'Content-Length': '0'} 2024-04-09T21:52:43.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000531297b9870523ed-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.313 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.313 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.314 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.314 DEBUG:boto:Path: /xryqqk-11/obj4 2024-04-09T21:52:43.314 DEBUG:boto:Data: 2024-04-09T21:52:43.314 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.314 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.314 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.314 DEBUG:boto:Params: {} 2024-04-09T21:52:43.314 DEBUG:boto:Token: None 2024-04-09T21:52:43.314 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4 2024-04-09T21:52:43.314 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE= 2024-04-09T21:52:43.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE=', 'Content-Length': '0'} 2024-04-09T21:52:43.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48a86fd3321d638-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-09T21:52:43.316 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.316 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.317 DEBUG:boto:Method: GET 2024-04-09T21:52:43.317 DEBUG:boto:Path: /xryqqk-11/obj4 2024-04-09T21:52:43.317 DEBUG:boto:Data: 2024-04-09T21:52:43.317 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.317 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.317 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.317 DEBUG:boto:Params: {} 2024-04-09T21:52:43.317 DEBUG:boto:Token: None 2024-04-09T21:52:43.317 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4 2024-04-09T21:52:43.317 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA= 2024-04-09T21:52:43.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA=', 'Content-Length': '0'} 2024-04-09T21:52:43.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b5b4ff62288c77-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.362 DEBUG:boto:path=/xryqqk-11/obj4 2024-04-09T21:52:43.363 DEBUG:boto:auth_path=/xryqqk-11/obj4 2024-04-09T21:52:43.363 DEBUG:boto:Method: GET 2024-04-09T21:52:43.363 DEBUG:boto:Path: /xryqqk-11/obj4 2024-04-09T21:52:43.363 DEBUG:boto:Data: 2024-04-09T21:52:43.363 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.363 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.363 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.363 DEBUG:boto:Params: {} 2024-04-09T21:52:43.363 DEBUG:boto:Token: None 2024-04-09T21:52:43.363 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj4 2024-04-09T21:52:43.364 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA= 2024-04-09T21:52:43.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA=', 'Content-Length': '0'} 2024-04-09T21:52:43.365 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055b1029cd98276e0-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.366 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.366 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.366 DEBUG:boto:path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.366 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.366 DEBUG:boto:Method: GET 2024-04-09T21:52:43.366 DEBUG:boto:Path: /xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.366 DEBUG:boto:Data: 2024-04-09T21:52:43.367 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.367 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.367 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.367 DEBUG:boto:Params: {} 2024-04-09T21:52:43.367 DEBUG:boto:Token: None 2024-04-09T21:52:43.367 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.367 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d63faaea8559f9-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.370 DEBUG:boto:b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.371 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.371 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.371 DEBUG:boto:path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.371 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.371 DEBUG:boto:Method: GET 2024-04-09T21:52:43.371 DEBUG:boto:Path: /xryqqk-11/?prefix=obj4 2024-04-09T21:52:43.371 DEBUG:boto:Data: 2024-04-09T21:52:43.371 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.371 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.371 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.371 DEBUG:boto:Params: {} 2024-04-09T21:52:43.372 DEBUG:boto:Token: None 2024-04-09T21:52:43.372 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.372 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007310ae2a4bf3d60e-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.375 DEBUG:boto:b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-04-09T21:52:43.376 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.376 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.376 DEBUG:boto:path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.376 DEBUG:boto:auth_path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.376 DEBUG:boto:Method: GET 2024-04-09T21:52:43.376 DEBUG:boto:Path: /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.376 DEBUG:boto:Data: 2024-04-09T21:52:43.376 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.377 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.377 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.377 DEBUG:boto:Params: {} 2024-04-09T21:52:43.377 DEBUG:boto:Token: None 2024-04-09T21:52:43.377 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.377 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-09T21:52:43.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-09T21:52:43.379 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005acc8c4ceaed58fd-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.380 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.380 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.380 DEBUG:boto:path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.380 DEBUG:boto:auth_path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.380 DEBUG:boto:Method: GET 2024-04-09T21:52:43.380 DEBUG:boto:Path: /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.380 DEBUG:boto:Data: 2024-04-09T21:52:43.380 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.381 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.381 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.381 DEBUG:boto:Params: {} 2024-04-09T21:52:43.381 DEBUG:boto:Token: None 2024-04-09T21:52:43.381 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.381 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-09T21:52:43.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-09T21:52:43.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b6d7d68b4d4c233-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.383 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.383 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.384 DEBUG:boto:path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.384 DEBUG:boto:auth_path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.384 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.384 DEBUG:boto:Path: /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.384 DEBUG:boto:Data: 2024-04-09T21:52:43.384 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.384 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.384 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.384 DEBUG:boto:Params: {} 2024-04-09T21:52:43.384 DEBUG:boto:Token: None 2024-04-09T21:52:43.384 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.385 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8= 2024-04-09T21:52:43.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8=', 'Content-Length': '0'} 2024-04-09T21:52:43.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002196d2a2e362b038-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.387 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.387 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.387 DEBUG:boto:path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.388 DEBUG:boto:auth_path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.388 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.388 DEBUG:boto:Path: /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.388 DEBUG:boto:Data: 2024-04-09T21:52:43.388 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.388 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.388 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.388 DEBUG:boto:Params: {} 2024-04-09T21:52:43.388 DEBUG:boto:Token: None 2024-04-09T21:52:43.388 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.389 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8= 2024-04-09T21:52:43.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8=', 'Content-Length': '0'} 2024-04-09T21:52:43.390 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000494c5e6144603cd1-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-04-09T21:52:43.391 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.391 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.391 DEBUG:boto:path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.391 DEBUG:boto:auth_path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.391 DEBUG:boto:Method: GET 2024-04-09T21:52:43.391 DEBUG:boto:Path: /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.391 DEBUG:boto:Data: 2024-04-09T21:52:43.391 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.391 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.391 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.391 DEBUG:boto:Params: {} 2024-04-09T21:52:43.392 DEBUG:boto:Token: None 2024-04-09T21:52:43.392 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.392 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-09T21:52:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-09T21:52:43.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59898f9f466dd50-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.393 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.394 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.394 DEBUG:boto:path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.394 DEBUG:boto:auth_path=/xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.394 DEBUG:boto:Method: GET 2024-04-09T21:52:43.394 DEBUG:boto:Path: /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.394 DEBUG:boto:Data: 2024-04-09T21:52:43.394 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.394 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.394 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.394 DEBUG:boto:Params: {} 2024-04-09T21:52:43.395 DEBUG:boto:Token: None 2024-04-09T21:52:43.395 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40?versionId=null 2024-04-09T21:52:43.395 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-09T21:52:43.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-09T21:52:43.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eb56693d3ef1c95-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.397 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.397 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.397 DEBUG:boto:path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.397 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.397 DEBUG:boto:Method: GET 2024-04-09T21:52:43.397 DEBUG:boto:Path: /xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.397 DEBUG:boto:Data: 2024-04-09T21:52:43.397 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.397 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.397 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.398 DEBUG:boto:Params: {} 2024-04-09T21:52:43.398 DEBUG:boto:Token: None 2024-04-09T21:52:43.398 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.398 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b32819dd4f4edc-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.400 DEBUG:boto:b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.401 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.401 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.401 DEBUG:boto:path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.401 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.401 DEBUG:boto:Method: GET 2024-04-09T21:52:43.401 DEBUG:boto:Path: /xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.401 DEBUG:boto:Data: 2024-04-09T21:52:43.401 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.401 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.401 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.401 DEBUG:boto:Params: {} 2024-04-09T21:52:43.402 DEBUG:boto:Token: None 2024-04-09T21:52:43.402 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.402 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079361c2e3a11b584-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.404 DEBUG:boto:b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.404 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.405 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.405 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.405 DEBUG:boto:Path: /xryqqk-11/obj40 2024-04-09T21:52:43.405 DEBUG:boto:Data: 2024-04-09T21:52:43.405 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.405 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.405 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.405 DEBUG:boto:Params: {} 2024-04-09T21:52:43.405 DEBUG:boto:Token: None 2024-04-09T21:52:43.405 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40 2024-04-09T21:52:43.406 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w= 2024-04-09T21:52:43.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w=', 'Content-Length': '0'} 2024-04-09T21:52:43.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f4b0ceb134eb0bc-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.407 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.408 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.408 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.408 DEBUG:boto:Path: /xryqqk-11/obj40 2024-04-09T21:52:43.408 DEBUG:boto:Data: 2024-04-09T21:52:43.408 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.408 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.408 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.408 DEBUG:boto:Params: {} 2024-04-09T21:52:43.408 DEBUG:boto:Token: None 2024-04-09T21:52:43.409 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40 2024-04-09T21:52:43.409 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w= 2024-04-09T21:52:43.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w=', 'Content-Length': '0'} 2024-04-09T21:52:43.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000577607b3ae10b657-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj40 2024-04-09T21:52:43.410 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.411 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.411 DEBUG:boto:Method: GET 2024-04-09T21:52:43.411 DEBUG:boto:Path: /xryqqk-11/obj40 2024-04-09T21:52:43.411 DEBUG:boto:Data: 2024-04-09T21:52:43.411 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.411 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.411 DEBUG:boto:Params: {} 2024-04-09T21:52:43.411 DEBUG:boto:Token: None 2024-04-09T21:52:43.411 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40 2024-04-09T21:52:43.412 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI= 2024-04-09T21:52:43.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI=', 'Content-Length': '0'} 2024-04-09T21:52:43.413 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b722cb1dbe26c71-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.413 DEBUG:boto:path=/xryqqk-11/obj40 2024-04-09T21:52:43.414 DEBUG:boto:auth_path=/xryqqk-11/obj40 2024-04-09T21:52:43.414 DEBUG:boto:Method: GET 2024-04-09T21:52:43.414 DEBUG:boto:Path: /xryqqk-11/obj40 2024-04-09T21:52:43.414 DEBUG:boto:Data: 2024-04-09T21:52:43.414 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.414 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.414 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.414 DEBUG:boto:Params: {} 2024-04-09T21:52:43.414 DEBUG:boto:Token: None 2024-04-09T21:52:43.415 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj40 2024-04-09T21:52:43.415 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI= 2024-04-09T21:52:43.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI=', 'Content-Length': '0'} 2024-04-09T21:52:43.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c1188aa7635e26-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.416 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.417 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.417 DEBUG:boto:path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.417 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.417 DEBUG:boto:Method: GET 2024-04-09T21:52:43.417 DEBUG:boto:Path: /xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.417 DEBUG:boto:Data: 2024-04-09T21:52:43.417 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.417 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.417 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.417 DEBUG:boto:Params: {} 2024-04-09T21:52:43.417 DEBUG:boto:Token: None 2024-04-09T21:52:43.418 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.418 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5226e971a8e9758-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.420 DEBUG:boto:b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.420 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.421 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.421 DEBUG:boto:path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.421 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.421 DEBUG:boto:Method: GET 2024-04-09T21:52:43.421 DEBUG:boto:Path: /xryqqk-11/?prefix=obj40 2024-04-09T21:52:43.421 DEBUG:boto:Data: 2024-04-09T21:52:43.421 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.421 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.421 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.421 DEBUG:boto:Params: {} 2024-04-09T21:52:43.422 DEBUG:boto:Token: None 2024-04-09T21:52:43.422 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.422 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.424 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceaa7f37fc65d2a9-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.424 DEBUG:boto:b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-04-09T21:52:43.425 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.425 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.425 DEBUG:boto:path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.425 DEBUG:boto:auth_path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.425 DEBUG:boto:Method: GET 2024-04-09T21:52:43.425 DEBUG:boto:Path: /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.425 DEBUG:boto:Data: 2024-04-09T21:52:43.425 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.425 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.425 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.425 DEBUG:boto:Params: {} 2024-04-09T21:52:43.426 DEBUG:boto:Token: None 2024-04-09T21:52:43.426 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.426 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-09T21:52:43.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-09T21:52:43.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ffa76a1691c583a-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.428 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.428 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.428 DEBUG:boto:path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.428 DEBUG:boto:auth_path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.429 DEBUG:boto:Method: GET 2024-04-09T21:52:43.429 DEBUG:boto:Path: /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.429 DEBUG:boto:Data: 2024-04-09T21:52:43.429 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.429 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.430 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.430 DEBUG:boto:Params: {} 2024-04-09T21:52:43.430 DEBUG:boto:Token: None 2024-04-09T21:52:43.430 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.430 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-09T21:52:43.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-09T21:52:43.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65a3919a14ffa3a-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.432 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.432 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.432 DEBUG:boto:path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.432 DEBUG:boto:auth_path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.432 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.432 DEBUG:boto:Path: /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.433 DEBUG:boto:Data: 2024-04-09T21:52:43.433 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.433 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.433 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.433 DEBUG:boto:Params: {} 2024-04-09T21:52:43.433 DEBUG:boto:Token: None 2024-04-09T21:52:43.433 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.434 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU= 2024-04-09T21:52:43.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU=', 'Content-Length': '0'} 2024-04-09T21:52:43.435 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbcb8c25f191c965-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.435 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.436 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.436 DEBUG:boto:path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.436 DEBUG:boto:auth_path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.436 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.436 DEBUG:boto:Path: /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.436 DEBUG:boto:Data: 2024-04-09T21:52:43.436 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.436 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.436 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.436 DEBUG:boto:Params: {} 2024-04-09T21:52:43.436 DEBUG:boto:Token: None 2024-04-09T21:52:43.437 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.437 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU= 2024-04-09T21:52:43.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU=', 'Content-Length': '0'} 2024-04-09T21:52:43.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050a29af3044c7c8c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-04-09T21:52:43.439 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.439 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.439 DEBUG:boto:path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.439 DEBUG:boto:auth_path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.439 DEBUG:boto:Method: GET 2024-04-09T21:52:43.439 DEBUG:boto:Path: /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.439 DEBUG:boto:Data: 2024-04-09T21:52:43.439 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.439 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.439 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.439 DEBUG:boto:Params: {} 2024-04-09T21:52:43.440 DEBUG:boto:Token: None 2024-04-09T21:52:43.440 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.440 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-09T21:52:43.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-09T21:52:43.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b309951a28142c59-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.442 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.442 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.442 DEBUG:boto:path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.442 DEBUG:boto:auth_path=/xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.442 DEBUG:boto:Method: GET 2024-04-09T21:52:43.443 DEBUG:boto:Path: /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.443 DEBUG:boto:Data: 2024-04-09T21:52:43.443 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.443 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.443 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.443 DEBUG:boto:Params: {} 2024-04-09T21:52:43.443 DEBUG:boto:Token: None 2024-04-09T21:52:43.444 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41?versionId=null 2024-04-09T21:52:43.444 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-09T21:52:43.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-09T21:52:43.445 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086b920ed2d5f213d-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.445 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.446 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.446 DEBUG:boto:path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.446 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.446 DEBUG:boto:Method: GET 2024-04-09T21:52:43.446 DEBUG:boto:Path: /xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.446 DEBUG:boto:Data: 2024-04-09T21:52:43.446 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.446 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.446 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.446 DEBUG:boto:Params: {} 2024-04-09T21:52:43.446 DEBUG:boto:Token: None 2024-04-09T21:52:43.447 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.447 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab28008771d83a6b-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.449 DEBUG:boto:b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.449 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.450 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.450 DEBUG:boto:path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.450 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.450 DEBUG:boto:Method: GET 2024-04-09T21:52:43.450 DEBUG:boto:Path: /xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.450 DEBUG:boto:Data: 2024-04-09T21:52:43.450 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.450 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.450 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.450 DEBUG:boto:Params: {} 2024-04-09T21:52:43.451 DEBUG:boto:Token: None 2024-04-09T21:52:43.451 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.451 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067132e261c2a7af4-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.453 DEBUG:boto:b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.453 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.453 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.454 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.454 DEBUG:boto:Path: /xryqqk-11/obj41 2024-04-09T21:52:43.454 DEBUG:boto:Data: 2024-04-09T21:52:43.454 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.454 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.454 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.454 DEBUG:boto:Params: {} 2024-04-09T21:52:43.454 DEBUG:boto:Token: None 2024-04-09T21:52:43.454 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41 2024-04-09T21:52:43.455 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA= 2024-04-09T21:52:43.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA=', 'Content-Length': '0'} 2024-04-09T21:52:43.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000d1769c6c3e5f94-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.456 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.456 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.457 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.457 DEBUG:boto:Path: /xryqqk-11/obj41 2024-04-09T21:52:43.457 DEBUG:boto:Data: 2024-04-09T21:52:43.457 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.457 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.457 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.457 DEBUG:boto:Params: {} 2024-04-09T21:52:43.457 DEBUG:boto:Token: None 2024-04-09T21:52:43.457 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41 2024-04-09T21:52:43.457 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA= 2024-04-09T21:52:43.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA=', 'Content-Length': '0'} 2024-04-09T21:52:43.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000babc05c1b14ebccf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj41 2024-04-09T21:52:43.460 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.460 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.460 DEBUG:boto:Method: GET 2024-04-09T21:52:43.460 DEBUG:boto:Path: /xryqqk-11/obj41 2024-04-09T21:52:43.460 DEBUG:boto:Data: 2024-04-09T21:52:43.460 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.460 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.460 DEBUG:boto:Params: {} 2024-04-09T21:52:43.461 DEBUG:boto:Token: None 2024-04-09T21:52:43.461 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41 2024-04-09T21:52:43.461 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI= 2024-04-09T21:52:43.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI=', 'Content-Length': '0'} 2024-04-09T21:52:43.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dacf7ca5ecc66aea-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.462 DEBUG:boto:path=/xryqqk-11/obj41 2024-04-09T21:52:43.463 DEBUG:boto:auth_path=/xryqqk-11/obj41 2024-04-09T21:52:43.463 DEBUG:boto:Method: GET 2024-04-09T21:52:43.463 DEBUG:boto:Path: /xryqqk-11/obj41 2024-04-09T21:52:43.463 DEBUG:boto:Data: 2024-04-09T21:52:43.463 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.463 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.463 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.463 DEBUG:boto:Params: {} 2024-04-09T21:52:43.463 DEBUG:boto:Token: None 2024-04-09T21:52:43.463 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj41 2024-04-09T21:52:43.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI= 2024-04-09T21:52:43.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI=', 'Content-Length': '0'} 2024-04-09T21:52:43.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093e605ac75674c9c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.510 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.511 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.511 DEBUG:boto:path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.511 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.511 DEBUG:boto:Method: GET 2024-04-09T21:52:43.511 DEBUG:boto:Path: /xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.511 DEBUG:boto:Data: 2024-04-09T21:52:43.511 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.511 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.511 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.511 DEBUG:boto:Params: {} 2024-04-09T21:52:43.512 DEBUG:boto:Token: None 2024-04-09T21:52:43.512 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.512 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da059099dec481e-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.514 DEBUG:boto:b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.515 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.515 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.515 DEBUG:boto:path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.515 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.515 DEBUG:boto:Method: GET 2024-04-09T21:52:43.515 DEBUG:boto:Path: /xryqqk-11/?prefix=obj41 2024-04-09T21:52:43.515 DEBUG:boto:Data: 2024-04-09T21:52:43.516 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.516 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.516 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.516 DEBUG:boto:Params: {} 2024-04-09T21:52:43.516 DEBUG:boto:Token: None 2024-04-09T21:52:43.516 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.516 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787486faf2c3395a-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.520 DEBUG:boto:b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-04-09T21:52:43.520 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.520 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.520 DEBUG:boto:path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.520 DEBUG:boto:auth_path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.521 DEBUG:boto:Method: GET 2024-04-09T21:52:43.521 DEBUG:boto:Path: /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.521 DEBUG:boto:Data: 2024-04-09T21:52:43.521 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.521 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.521 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.521 DEBUG:boto:Params: {} 2024-04-09T21:52:43.521 DEBUG:boto:Token: None 2024-04-09T21:52:43.521 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.521 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-09T21:52:43.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-09T21:52:43.523 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5886c5f291ab61-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.566 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.567 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.567 DEBUG:boto:path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.567 DEBUG:boto:auth_path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.567 DEBUG:boto:Method: GET 2024-04-09T21:52:43.567 DEBUG:boto:Path: /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.567 DEBUG:boto:Data: 2024-04-09T21:52:43.567 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.567 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.567 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.567 DEBUG:boto:Params: {} 2024-04-09T21:52:43.568 DEBUG:boto:Token: None 2024-04-09T21:52:43.568 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.568 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-09T21:52:43.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-09T21:52:43.569 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aef2171073ed7b7-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.570 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.570 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.570 DEBUG:boto:path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.570 DEBUG:boto:auth_path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.570 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.570 DEBUG:boto:Path: /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.570 DEBUG:boto:Data: 2024-04-09T21:52:43.570 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.570 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.571 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.571 DEBUG:boto:Params: {} 2024-04-09T21:52:43.571 DEBUG:boto:Token: None 2024-04-09T21:52:43.571 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.571 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q= 2024-04-09T21:52:43.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q=', 'Content-Length': '0'} 2024-04-09T21:52:43.573 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064d04266b62f54c7-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.573 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.573 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.573 DEBUG:boto:path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.573 DEBUG:boto:auth_path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.573 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.573 DEBUG:boto:Path: /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.573 DEBUG:boto:Data: 2024-04-09T21:52:43.573 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.574 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.574 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.574 DEBUG:boto:Params: {} 2024-04-09T21:52:43.574 DEBUG:boto:Token: None 2024-04-09T21:52:43.574 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.574 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q= 2024-04-09T21:52:43.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q=', 'Content-Length': '0'} 2024-04-09T21:52:43.576 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000814f4be9f4c6be2c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-04-09T21:52:43.576 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.576 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.576 DEBUG:boto:path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.576 DEBUG:boto:auth_path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.576 DEBUG:boto:Method: GET 2024-04-09T21:52:43.577 DEBUG:boto:Path: /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.577 DEBUG:boto:Data: 2024-04-09T21:52:43.577 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.577 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.577 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.577 DEBUG:boto:Params: {} 2024-04-09T21:52:43.577 DEBUG:boto:Token: None 2024-04-09T21:52:43.577 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.577 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-09T21:52:43.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-09T21:52:43.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000197e601189b1155b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.579 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.579 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.580 DEBUG:boto:path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.580 DEBUG:boto:auth_path=/xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.580 DEBUG:boto:Method: GET 2024-04-09T21:52:43.580 DEBUG:boto:Path: /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.580 DEBUG:boto:Data: 2024-04-09T21:52:43.580 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.580 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.580 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.580 DEBUG:boto:Params: {} 2024-04-09T21:52:43.580 DEBUG:boto:Token: None 2024-04-09T21:52:43.581 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42?versionId=null 2024-04-09T21:52:43.581 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-09T21:52:43.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-09T21:52:43.582 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b22000df5db56301-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.582 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.582 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.583 DEBUG:boto:path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.583 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.583 DEBUG:boto:Method: GET 2024-04-09T21:52:43.583 DEBUG:boto:Path: /xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.583 DEBUG:boto:Data: 2024-04-09T21:52:43.583 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.583 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.583 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.583 DEBUG:boto:Params: {} 2024-04-09T21:52:43.583 DEBUG:boto:Token: None 2024-04-09T21:52:43.584 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.584 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dab7d5294b57c24-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.586 DEBUG:boto:b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.586 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.586 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.586 DEBUG:boto:path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.586 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.587 DEBUG:boto:Method: GET 2024-04-09T21:52:43.587 DEBUG:boto:Path: /xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.587 DEBUG:boto:Data: 2024-04-09T21:52:43.587 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.587 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.587 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.587 DEBUG:boto:Params: {} 2024-04-09T21:52:43.587 DEBUG:boto:Token: None 2024-04-09T21:52:43.587 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.587 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0aee3b7f1187224-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.590 DEBUG:boto:b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.590 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.590 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.590 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.590 DEBUG:boto:Path: /xryqqk-11/obj42 2024-04-09T21:52:43.591 DEBUG:boto:Data: 2024-04-09T21:52:43.591 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.591 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.591 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.591 DEBUG:boto:Params: {} 2024-04-09T21:52:43.591 DEBUG:boto:Token: None 2024-04-09T21:52:43.591 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42 2024-04-09T21:52:43.591 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk= 2024-04-09T21:52:43.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk=', 'Content-Length': '0'} 2024-04-09T21:52:43.593 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045d1d5418ec977a5-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.593 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.593 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.593 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.593 DEBUG:boto:Path: /xryqqk-11/obj42 2024-04-09T21:52:43.593 DEBUG:boto:Data: 2024-04-09T21:52:43.593 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.594 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.594 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.594 DEBUG:boto:Params: {} 2024-04-09T21:52:43.594 DEBUG:boto:Token: None 2024-04-09T21:52:43.594 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42 2024-04-09T21:52:43.594 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk= 2024-04-09T21:52:43.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk=', 'Content-Length': '0'} 2024-04-09T21:52:43.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fe9f30210faf905-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj42 2024-04-09T21:52:43.596 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.596 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.596 DEBUG:boto:Method: GET 2024-04-09T21:52:43.596 DEBUG:boto:Path: /xryqqk-11/obj42 2024-04-09T21:52:43.596 DEBUG:boto:Data: 2024-04-09T21:52:43.596 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.596 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.597 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.597 DEBUG:boto:Params: {} 2024-04-09T21:52:43.597 DEBUG:boto:Token: None 2024-04-09T21:52:43.597 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42 2024-04-09T21:52:43.597 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM= 2024-04-09T21:52:43.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM=', 'Content-Length': '0'} 2024-04-09T21:52:43.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000751c6cc5e5e2839a-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.599 DEBUG:boto:path=/xryqqk-11/obj42 2024-04-09T21:52:43.600 DEBUG:boto:auth_path=/xryqqk-11/obj42 2024-04-09T21:52:43.600 DEBUG:boto:Method: GET 2024-04-09T21:52:43.600 DEBUG:boto:Path: /xryqqk-11/obj42 2024-04-09T21:52:43.600 DEBUG:boto:Data: 2024-04-09T21:52:43.600 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.600 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.600 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.600 DEBUG:boto:Params: {} 2024-04-09T21:52:43.601 DEBUG:boto:Token: None 2024-04-09T21:52:43.601 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj42 2024-04-09T21:52:43.601 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM= 2024-04-09T21:52:43.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM=', 'Content-Length': '0'} 2024-04-09T21:52:43.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aef050f28ae63e3a-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.603 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.604 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.604 DEBUG:boto:path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.604 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.604 DEBUG:boto:Method: GET 2024-04-09T21:52:43.604 DEBUG:boto:Path: /xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.604 DEBUG:boto:Data: 2024-04-09T21:52:43.604 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.604 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.604 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.604 DEBUG:boto:Params: {} 2024-04-09T21:52:43.605 DEBUG:boto:Token: None 2024-04-09T21:52:43.605 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.605 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa2b0f125958c11-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.654 DEBUG:boto:b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.655 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.655 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.655 DEBUG:boto:path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.655 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.655 DEBUG:boto:Method: GET 2024-04-09T21:52:43.656 DEBUG:boto:Path: /xryqqk-11/?prefix=obj42 2024-04-09T21:52:43.656 DEBUG:boto:Data: 2024-04-09T21:52:43.656 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.656 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.656 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.656 DEBUG:boto:Params: {} 2024-04-09T21:52:43.656 DEBUG:boto:Token: None 2024-04-09T21:52:43.656 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bcb3ce488eaf2e5-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.659 DEBUG:boto:b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-04-09T21:52:43.659 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.659 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.660 DEBUG:boto:path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.660 DEBUG:boto:auth_path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.660 DEBUG:boto:Method: GET 2024-04-09T21:52:43.660 DEBUG:boto:Path: /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.660 DEBUG:boto:Data: 2024-04-09T21:52:43.660 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.660 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.660 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.660 DEBUG:boto:Params: {} 2024-04-09T21:52:43.660 DEBUG:boto:Token: None 2024-04-09T21:52:43.660 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.661 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-09T21:52:43.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-09T21:52:43.662 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c1d6ff890d0e298-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.662 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.663 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.663 DEBUG:boto:path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.663 DEBUG:boto:auth_path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.663 DEBUG:boto:Method: GET 2024-04-09T21:52:43.663 DEBUG:boto:Path: /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.663 DEBUG:boto:Data: 2024-04-09T21:52:43.663 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.663 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.663 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.663 DEBUG:boto:Params: {} 2024-04-09T21:52:43.663 DEBUG:boto:Token: None 2024-04-09T21:52:43.664 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-09T21:52:43.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-09T21:52:43.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000882f776e27d82707-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.665 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.665 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.666 DEBUG:boto:path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.666 DEBUG:boto:auth_path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.666 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.666 DEBUG:boto:Path: /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.666 DEBUG:boto:Data: 2024-04-09T21:52:43.666 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.666 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.666 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.666 DEBUG:boto:Params: {} 2024-04-09T21:52:43.666 DEBUG:boto:Token: None 2024-04-09T21:52:43.667 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.667 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI= 2024-04-09T21:52:43.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI=', 'Content-Length': '0'} 2024-04-09T21:52:43.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009522b2c4dd84e076-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.669 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.669 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.669 DEBUG:boto:path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.669 DEBUG:boto:auth_path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.669 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.669 DEBUG:boto:Path: /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.669 DEBUG:boto:Data: 2024-04-09T21:52:43.669 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.669 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.669 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.669 DEBUG:boto:Params: {} 2024-04-09T21:52:43.670 DEBUG:boto:Token: None 2024-04-09T21:52:43.670 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.670 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI= 2024-04-09T21:52:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI=', 'Content-Length': '0'} 2024-04-09T21:52:43.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097fe60549ad57d5f-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-04-09T21:52:43.672 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.672 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.672 DEBUG:boto:path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.672 DEBUG:boto:auth_path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.672 DEBUG:boto:Method: GET 2024-04-09T21:52:43.672 DEBUG:boto:Path: /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.672 DEBUG:boto:Data: 2024-04-09T21:52:43.672 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.672 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.673 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.673 DEBUG:boto:Params: {} 2024-04-09T21:52:43.673 DEBUG:boto:Token: None 2024-04-09T21:52:43.673 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.673 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-09T21:52:43.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-09T21:52:43.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ce699e15b673efa-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.675 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.675 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.675 DEBUG:boto:path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.675 DEBUG:boto:auth_path=/xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.675 DEBUG:boto:Method: GET 2024-04-09T21:52:43.675 DEBUG:boto:Path: /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.676 DEBUG:boto:Data: 2024-04-09T21:52:43.676 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.676 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.676 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.676 DEBUG:boto:Params: {} 2024-04-09T21:52:43.676 DEBUG:boto:Token: None 2024-04-09T21:52:43.676 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43?versionId=null 2024-04-09T21:52:43.676 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-09T21:52:43.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-09T21:52:43.678 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7a91b3f1fd6ed61-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.678 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.678 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.678 DEBUG:boto:path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.679 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.679 DEBUG:boto:Method: GET 2024-04-09T21:52:43.679 DEBUG:boto:Path: /xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.679 DEBUG:boto:Data: 2024-04-09T21:52:43.679 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.679 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.679 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.679 DEBUG:boto:Params: {} 2024-04-09T21:52:43.679 DEBUG:boto:Token: None 2024-04-09T21:52:43.679 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.680 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0e9902488490a4b-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.722 DEBUG:boto:b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.723 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.723 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.723 DEBUG:boto:path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.723 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.723 DEBUG:boto:Method: GET 2024-04-09T21:52:43.723 DEBUG:boto:Path: /xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.723 DEBUG:boto:Data: 2024-04-09T21:52:43.723 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.724 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.724 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.724 DEBUG:boto:Params: {} 2024-04-09T21:52:43.724 DEBUG:boto:Token: None 2024-04-09T21:52:43.724 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.724 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d683728346dfb3-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.727 DEBUG:boto:b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.727 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.727 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.727 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.727 DEBUG:boto:Path: /xryqqk-11/obj43 2024-04-09T21:52:43.727 DEBUG:boto:Data: 2024-04-09T21:52:43.727 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.727 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.728 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.728 DEBUG:boto:Params: {} 2024-04-09T21:52:43.728 DEBUG:boto:Token: None 2024-04-09T21:52:43.728 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43 2024-04-09T21:52:43.728 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw= 2024-04-09T21:52:43.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw=', 'Content-Length': '0'} 2024-04-09T21:52:43.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fb69a31fbb07a38-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.730 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.730 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.730 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.730 DEBUG:boto:Path: /xryqqk-11/obj43 2024-04-09T21:52:43.730 DEBUG:boto:Data: 2024-04-09T21:52:43.730 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.730 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.730 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.730 DEBUG:boto:Params: {} 2024-04-09T21:52:43.730 DEBUG:boto:Token: None 2024-04-09T21:52:43.731 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43 2024-04-09T21:52:43.731 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw= 2024-04-09T21:52:43.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw=', 'Content-Length': '0'} 2024-04-09T21:52:43.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000252835fa513f31ac-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj43 2024-04-09T21:52:43.733 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.733 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.733 DEBUG:boto:Method: GET 2024-04-09T21:52:43.733 DEBUG:boto:Path: /xryqqk-11/obj43 2024-04-09T21:52:43.733 DEBUG:boto:Data: 2024-04-09T21:52:43.733 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.733 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.733 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.733 DEBUG:boto:Params: {} 2024-04-09T21:52:43.733 DEBUG:boto:Token: None 2024-04-09T21:52:43.734 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43 2024-04-09T21:52:43.734 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE= 2024-04-09T21:52:43.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE=', 'Content-Length': '0'} 2024-04-09T21:52:43.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073992b58cb76d18b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.735 DEBUG:boto:path=/xryqqk-11/obj43 2024-04-09T21:52:43.736 DEBUG:boto:auth_path=/xryqqk-11/obj43 2024-04-09T21:52:43.736 DEBUG:boto:Method: GET 2024-04-09T21:52:43.736 DEBUG:boto:Path: /xryqqk-11/obj43 2024-04-09T21:52:43.736 DEBUG:boto:Data: 2024-04-09T21:52:43.736 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.736 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.736 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.736 DEBUG:boto:Params: {} 2024-04-09T21:52:43.736 DEBUG:boto:Token: None 2024-04-09T21:52:43.736 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj43 2024-04-09T21:52:43.737 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE= 2024-04-09T21:52:43.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE=', 'Content-Length': '0'} 2024-04-09T21:52:43.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e901e474d2776850-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.778 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.779 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.779 DEBUG:boto:path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.779 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.779 DEBUG:boto:Method: GET 2024-04-09T21:52:43.779 DEBUG:boto:Path: /xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.779 DEBUG:boto:Data: 2024-04-09T21:52:43.779 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.779 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.779 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.779 DEBUG:boto:Params: {} 2024-04-09T21:52:43.780 DEBUG:boto:Token: None 2024-04-09T21:52:43.780 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000700ff1e60d6659ac-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.782 DEBUG:boto:b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.783 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.783 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.783 DEBUG:boto:path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.783 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.783 DEBUG:boto:Method: GET 2024-04-09T21:52:43.783 DEBUG:boto:Path: /xryqqk-11/?prefix=obj43 2024-04-09T21:52:43.783 DEBUG:boto:Data: 2024-04-09T21:52:43.784 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.784 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.784 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.784 DEBUG:boto:Params: {} 2024-04-09T21:52:43.784 DEBUG:boto:Token: None 2024-04-09T21:52:43.784 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.784 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7b594c725a233dc-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.786 DEBUG:boto:b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-09T21:52:43.787 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.787 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.787 DEBUG:boto:path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.787 DEBUG:boto:auth_path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.787 DEBUG:boto:Method: GET 2024-04-09T21:52:43.787 DEBUG:boto:Path: /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.787 DEBUG:boto:Data: 2024-04-09T21:52:43.788 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.788 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.788 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.788 DEBUG:boto:Params: {} 2024-04-09T21:52:43.788 DEBUG:boto:Token: None 2024-04-09T21:52:43.788 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.788 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-09T21:52:43.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-09T21:52:43.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9f072a70a30094f-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.790 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.790 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.791 DEBUG:boto:path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.791 DEBUG:boto:auth_path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.791 DEBUG:boto:Method: GET 2024-04-09T21:52:43.791 DEBUG:boto:Path: /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.791 DEBUG:boto:Data: 2024-04-09T21:52:43.791 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.791 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.791 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.791 DEBUG:boto:Params: {} 2024-04-09T21:52:43.791 DEBUG:boto:Token: None 2024-04-09T21:52:43.791 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.792 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-09T21:52:43.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-09T21:52:43.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048fd593cbde71273-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.794 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.794 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.794 DEBUG:boto:path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.794 DEBUG:boto:auth_path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.794 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.795 DEBUG:boto:Path: /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.795 DEBUG:boto:Data: 2024-04-09T21:52:43.795 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.795 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.795 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.795 DEBUG:boto:Params: {} 2024-04-09T21:52:43.795 DEBUG:boto:Token: None 2024-04-09T21:52:43.795 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.795 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc= 2024-04-09T21:52:43.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc=', 'Content-Length': '0'} 2024-04-09T21:52:43.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bd007a2fe0c02e3-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.797 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.797 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.797 DEBUG:boto:path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.797 DEBUG:boto:auth_path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.797 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.797 DEBUG:boto:Path: /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.798 DEBUG:boto:Data: 2024-04-09T21:52:43.798 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.798 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.798 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.798 DEBUG:boto:Params: {} 2024-04-09T21:52:43.798 DEBUG:boto:Token: None 2024-04-09T21:52:43.798 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.798 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc= 2024-04-09T21:52:43.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc=', 'Content-Length': '0'} 2024-04-09T21:52:43.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8940b62dca26242-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-09T21:52:43.800 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.800 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.800 DEBUG:boto:path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.800 DEBUG:boto:auth_path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.800 DEBUG:boto:Method: GET 2024-04-09T21:52:43.801 DEBUG:boto:Path: /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.801 DEBUG:boto:Data: 2024-04-09T21:52:43.801 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.801 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.801 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.801 DEBUG:boto:Params: {} 2024-04-09T21:52:43.801 DEBUG:boto:Token: None 2024-04-09T21:52:43.801 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.801 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-09T21:52:43.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-09T21:52:43.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078c10c7fb134e11b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.803 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.803 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.803 DEBUG:boto:path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.803 DEBUG:boto:auth_path=/xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.804 DEBUG:boto:Method: GET 2024-04-09T21:52:43.804 DEBUG:boto:Path: /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.804 DEBUG:boto:Data: 2024-04-09T21:52:43.804 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.804 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.804 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.804 DEBUG:boto:Params: {} 2024-04-09T21:52:43.804 DEBUG:boto:Token: None 2024-04-09T21:52:43.804 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44?versionId=null 2024-04-09T21:52:43.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-09T21:52:43.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-09T21:52:43.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007717b4418e1961d5-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.806 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.807 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.807 DEBUG:boto:path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.807 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.807 DEBUG:boto:Method: GET 2024-04-09T21:52:43.807 DEBUG:boto:Path: /xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.807 DEBUG:boto:Data: 2024-04-09T21:52:43.807 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.807 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.807 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.807 DEBUG:boto:Params: {} 2024-04-09T21:52:43.807 DEBUG:boto:Token: None 2024-04-09T21:52:43.808 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.808 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb10268c9692cb4-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.850 DEBUG:boto:b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.851 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.851 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.851 DEBUG:boto:path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.851 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.851 DEBUG:boto:Method: GET 2024-04-09T21:52:43.851 DEBUG:boto:Path: /xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.852 DEBUG:boto:Data: 2024-04-09T21:52:43.852 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.852 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.852 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.852 DEBUG:boto:Params: {} 2024-04-09T21:52:43.852 DEBUG:boto:Token: None 2024-04-09T21:52:43.852 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.852 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f85962ff8796273-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.856 DEBUG:boto:b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.856 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.856 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.856 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.856 DEBUG:boto:Path: /xryqqk-11/obj44 2024-04-09T21:52:43.857 DEBUG:boto:Data: 2024-04-09T21:52:43.857 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.857 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.857 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.857 DEBUG:boto:Params: {} 2024-04-09T21:52:43.857 DEBUG:boto:Token: None 2024-04-09T21:52:43.857 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44 2024-04-09T21:52:43.857 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas= 2024-04-09T21:52:43.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas=', 'Content-Length': '0'} 2024-04-09T21:52:43.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e40557f6a5dd2e-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.860 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.860 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.860 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.860 DEBUG:boto:Path: /xryqqk-11/obj44 2024-04-09T21:52:43.860 DEBUG:boto:Data: 2024-04-09T21:52:43.860 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.860 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.861 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.861 DEBUG:boto:Params: {} 2024-04-09T21:52:43.861 DEBUG:boto:Token: None 2024-04-09T21:52:43.861 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44 2024-04-09T21:52:43.861 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas= 2024-04-09T21:52:43.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas=', 'Content-Length': '0'} 2024-04-09T21:52:43.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c644c91e82d01a2d-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-09T21:52:43.864 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.864 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.864 DEBUG:boto:Method: GET 2024-04-09T21:52:43.865 DEBUG:boto:Path: /xryqqk-11/obj44 2024-04-09T21:52:43.865 DEBUG:boto:Data: 2024-04-09T21:52:43.865 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.865 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.865 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.865 DEBUG:boto:Params: {} 2024-04-09T21:52:43.865 DEBUG:boto:Token: None 2024-04-09T21:52:43.866 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44 2024-04-09T21:52:43.866 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc= 2024-04-09T21:52:43.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc=', 'Content-Length': '0'} 2024-04-09T21:52:43.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7ad537409d1e93f-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.868 DEBUG:boto:path=/xryqqk-11/obj44 2024-04-09T21:52:43.868 DEBUG:boto:auth_path=/xryqqk-11/obj44 2024-04-09T21:52:43.868 DEBUG:boto:Method: GET 2024-04-09T21:52:43.868 DEBUG:boto:Path: /xryqqk-11/obj44 2024-04-09T21:52:43.868 DEBUG:boto:Data: 2024-04-09T21:52:43.868 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.868 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.868 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.868 DEBUG:boto:Params: {} 2024-04-09T21:52:43.869 DEBUG:boto:Token: None 2024-04-09T21:52:43.869 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj44 2024-04-09T21:52:43.869 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc= 2024-04-09T21:52:43.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc=', 'Content-Length': '0'} 2024-04-09T21:52:43.870 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb737347e3608bfd-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.914 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.915 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.916 DEBUG:boto:path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.916 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.916 DEBUG:boto:Method: GET 2024-04-09T21:52:43.916 DEBUG:boto:Path: /xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.916 DEBUG:boto:Data: 2024-04-09T21:52:43.917 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.917 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.917 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.917 DEBUG:boto:Params: {} 2024-04-09T21:52:43.917 DEBUG:boto:Token: None 2024-04-09T21:52:43.917 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.918 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeeb7e2d11a74736-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.920 DEBUG:boto:b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.920 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.921 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.921 DEBUG:boto:path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.921 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.921 DEBUG:boto:Method: GET 2024-04-09T21:52:43.921 DEBUG:boto:Path: /xryqqk-11/?prefix=obj44 2024-04-09T21:52:43.921 DEBUG:boto:Data: 2024-04-09T21:52:43.921 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.921 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.921 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.921 DEBUG:boto:Params: {} 2024-04-09T21:52:43.921 DEBUG:boto:Token: None 2024-04-09T21:52:43.922 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.922 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdf7cddcb7fbe5a-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.924 DEBUG:boto:b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-04-09T21:52:43.925 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.925 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.925 DEBUG:boto:path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.925 DEBUG:boto:auth_path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.925 DEBUG:boto:Method: GET 2024-04-09T21:52:43.926 DEBUG:boto:Path: /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.926 DEBUG:boto:Data: 2024-04-09T21:52:43.926 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.926 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.926 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.926 DEBUG:boto:Params: {} 2024-04-09T21:52:43.926 DEBUG:boto:Token: None 2024-04-09T21:52:43.926 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.926 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-09T21:52:43.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-09T21:52:43.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000740e8f9affb39317-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.928 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.928 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.928 DEBUG:boto:path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.929 DEBUG:boto:auth_path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.929 DEBUG:boto:Method: GET 2024-04-09T21:52:43.929 DEBUG:boto:Path: /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.929 DEBUG:boto:Data: 2024-04-09T21:52:43.929 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.929 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.929 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.929 DEBUG:boto:Params: {} 2024-04-09T21:52:43.929 DEBUG:boto:Token: None 2024-04-09T21:52:43.929 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.929 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-09T21:52:43.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-09T21:52:43.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c50b02a1978ac1c9-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.931 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.931 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.931 DEBUG:boto:path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.932 DEBUG:boto:auth_path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.932 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.932 DEBUG:boto:Path: /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.932 DEBUG:boto:Data: 2024-04-09T21:52:43.932 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.932 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.932 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.932 DEBUG:boto:Params: {} 2024-04-09T21:52:43.932 DEBUG:boto:Token: None 2024-04-09T21:52:43.932 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.933 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ= 2024-04-09T21:52:43.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ=', 'Content-Length': '0'} 2024-04-09T21:52:43.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001db6e5e1ae817975-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.934 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.934 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.935 DEBUG:boto:path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.935 DEBUG:boto:auth_path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.935 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.935 DEBUG:boto:Path: /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.935 DEBUG:boto:Data: 2024-04-09T21:52:43.935 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.935 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.935 DEBUG:boto:Params: {} 2024-04-09T21:52:43.935 DEBUG:boto:Token: None 2024-04-09T21:52:43.936 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ= 2024-04-09T21:52:43.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ=', 'Content-Length': '0'} 2024-04-09T21:52:43.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ecb4d188f398c78-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-04-09T21:52:43.937 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.938 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.938 DEBUG:boto:path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.938 DEBUG:boto:auth_path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.938 DEBUG:boto:Method: GET 2024-04-09T21:52:43.938 DEBUG:boto:Path: /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.938 DEBUG:boto:Data: 2024-04-09T21:52:43.938 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.938 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.938 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.938 DEBUG:boto:Params: {} 2024-04-09T21:52:43.938 DEBUG:boto:Token: None 2024-04-09T21:52:43.939 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-09T21:52:43.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-09T21:52:43.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b76a68adf856159f-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.941 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.941 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.941 DEBUG:boto:path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.941 DEBUG:boto:auth_path=/xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.941 DEBUG:boto:Method: GET 2024-04-09T21:52:43.941 DEBUG:boto:Path: /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.941 DEBUG:boto:Data: 2024-04-09T21:52:43.941 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.941 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.941 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.942 DEBUG:boto:Params: {} 2024-04-09T21:52:43.942 DEBUG:boto:Token: None 2024-04-09T21:52:43.942 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45?versionId=null 2024-04-09T21:52:43.942 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-09T21:52:43.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-09T21:52:43.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fafd43da2718343-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.944 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.944 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.944 DEBUG:boto:path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:43.944 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:43.944 DEBUG:boto:Method: GET 2024-04-09T21:52:43.944 DEBUG:boto:Path: /xryqqk-11/?prefix=obj45 2024-04-09T21:52:43.945 DEBUG:boto:Data: 2024-04-09T21:52:43.945 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.945 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.945 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.945 DEBUG:boto:Params: {} 2024-04-09T21:52:43.945 DEBUG:boto:Token: None 2024-04-09T21:52:43.945 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.945 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c7ec34b7ac310c-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.947 DEBUG:boto:b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.948 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:43.948 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:43.948 DEBUG:boto:path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:43.948 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:43.948 DEBUG:boto:Method: GET 2024-04-09T21:52:43.949 DEBUG:boto:Path: /xryqqk-11/?prefix=obj45 2024-04-09T21:52:43.949 DEBUG:boto:Data: 2024-04-09T21:52:43.949 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.949 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.949 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.949 DEBUG:boto:Params: {} 2024-04-09T21:52:43.949 DEBUG:boto:Token: None 2024-04-09T21:52:43.949 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/ 2024-04-09T21:52:43.949 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-09T21:52:43.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-09T21:52:43.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007317b61567096103-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.951 DEBUG:boto:b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:43.952 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.952 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.952 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.952 DEBUG:boto:Path: /xryqqk-11/obj45 2024-04-09T21:52:43.952 DEBUG:boto:Data: 2024-04-09T21:52:43.952 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.952 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.952 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.952 DEBUG:boto:Params: {} 2024-04-09T21:52:43.953 DEBUG:boto:Token: None 2024-04-09T21:52:43.953 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45 2024-04-09T21:52:43.953 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI= 2024-04-09T21:52:43.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI=', 'Content-Length': '0'} 2024-04-09T21:52:43.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040bf1fe58492f1d9-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.954 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.955 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.955 DEBUG:boto:Method: HEAD 2024-04-09T21:52:43.955 DEBUG:boto:Path: /xryqqk-11/obj45 2024-04-09T21:52:43.955 DEBUG:boto:Data: 2024-04-09T21:52:43.955 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:43.955 DEBUG:boto:Port: 8000 2024-04-09T21:52:43.955 DEBUG:boto:Params: {} 2024-04-09T21:52:43.955 DEBUG:boto:Token: None 2024-04-09T21:52:43.955 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45 2024-04-09T21:52:43.955 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI= 2024-04-09T21:52:43.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI=', 'Content-Length': '0'} 2024-04-09T21:52:43.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d73d26fc82a843c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:43.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj45 2024-04-09T21:52:43.957 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:43.957 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:43.958 DEBUG:boto:Method: GET 2024-04-09T21:52:43.958 DEBUG:boto:Path: /xryqqk-11/obj45 2024-04-09T21:52:43.958 DEBUG:boto:Data: 2024-04-09T21:52:43.958 DEBUG:boto:Headers: {} 2024-04-09T21:52:43.958 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:43.958 DEBUG:boto:Port: 8001 2024-04-09T21:52:43.958 DEBUG:boto:Params: {} 2024-04-09T21:52:43.958 DEBUG:boto:Token: None 2024-04-09T21:52:43.958 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:43 GMT /xryqqk-11/obj45 2024-04-09T21:52:43.958 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:leQpFXMkn9G639FjF3AMs13u2Ko= 2024-04-09T21:52:43.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:leQpFXMkn9G639FjF3AMs13u2Ko=', 'Content-Length': '0'} 2024-04-09T21:52:43.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e26b3a1c551697-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.002 DEBUG:boto:path=/xryqqk-11/obj45 2024-04-09T21:52:44.003 DEBUG:boto:auth_path=/xryqqk-11/obj45 2024-04-09T21:52:44.003 DEBUG:boto:Method: GET 2024-04-09T21:52:44.003 DEBUG:boto:Path: /xryqqk-11/obj45 2024-04-09T21:52:44.003 DEBUG:boto:Data: 2024-04-09T21:52:44.003 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.003 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.003 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.003 DEBUG:boto:Params: {} 2024-04-09T21:52:44.003 DEBUG:boto:Token: None 2024-04-09T21:52:44.004 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj45 2024-04-09T21:52:44.004 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xKxiIkPHA9ioIiNE285KdoHqELQ= 2024-04-09T21:52:44.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xKxiIkPHA9ioIiNE285KdoHqELQ=', 'Content-Length': '0'} 2024-04-09T21:52:44.005 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a473a7aa383d2b03-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.006 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.006 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.006 DEBUG:boto:path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:44.006 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:44.006 DEBUG:boto:Method: GET 2024-04-09T21:52:44.006 DEBUG:boto:Path: /xryqqk-11/?prefix=obj45 2024-04-09T21:52:44.006 DEBUG:boto:Data: 2024-04-09T21:52:44.007 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.007 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.007 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.007 DEBUG:boto:Params: {} 2024-04-09T21:52:44.007 DEBUG:boto:Token: None 2024-04-09T21:52:44.007 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.007 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080fea8c619fc6b17-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.010 DEBUG:boto:b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.010 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.010 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.010 DEBUG:boto:path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:44.010 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj45 2024-04-09T21:52:44.010 DEBUG:boto:Method: GET 2024-04-09T21:52:44.010 DEBUG:boto:Path: /xryqqk-11/?prefix=obj45 2024-04-09T21:52:44.011 DEBUG:boto:Data: 2024-04-09T21:52:44.011 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.011 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.011 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.011 DEBUG:boto:Params: {} 2024-04-09T21:52:44.011 DEBUG:boto:Token: None 2024-04-09T21:52:44.011 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.011 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c4f2c08726f6c3-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.013 DEBUG:boto:b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-09T21:52:44.014 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.014 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.014 DEBUG:boto:path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.014 DEBUG:boto:auth_path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.014 DEBUG:boto:Method: GET 2024-04-09T21:52:44.014 DEBUG:boto:Path: /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.015 DEBUG:boto:Data: 2024-04-09T21:52:44.015 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.015 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.015 DEBUG:boto:Params: {} 2024-04-09T21:52:44.015 DEBUG:boto:Token: None 2024-04-09T21:52:44.015 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.015 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-09T21:52:44.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-09T21:52:44.017 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb6cf9fc116f97fe-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.018 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.018 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.018 DEBUG:boto:path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.018 DEBUG:boto:auth_path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.018 DEBUG:boto:Method: GET 2024-04-09T21:52:44.018 DEBUG:boto:Path: /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.018 DEBUG:boto:Data: 2024-04-09T21:52:44.018 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.018 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.019 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.019 DEBUG:boto:Params: {} 2024-04-09T21:52:44.019 DEBUG:boto:Token: None 2024-04-09T21:52:44.019 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.019 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-09T21:52:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-09T21:52:44.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057550b00c3381177-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.021 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.021 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.021 DEBUG:boto:path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.021 DEBUG:boto:auth_path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.021 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.021 DEBUG:boto:Path: /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.021 DEBUG:boto:Data: 2024-04-09T21:52:44.021 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.021 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.022 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.022 DEBUG:boto:Params: {} 2024-04-09T21:52:44.022 DEBUG:boto:Token: None 2024-04-09T21:52:44.022 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.022 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo= 2024-04-09T21:52:44.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo=', 'Content-Length': '0'} 2024-04-09T21:52:44.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d0d07e210cd8e6c-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.024 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.024 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.024 DEBUG:boto:path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.024 DEBUG:boto:auth_path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.024 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.024 DEBUG:boto:Path: /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.024 DEBUG:boto:Data: 2024-04-09T21:52:44.024 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.024 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.025 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.025 DEBUG:boto:Params: {} 2024-04-09T21:52:44.025 DEBUG:boto:Token: None 2024-04-09T21:52:44.025 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.025 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo= 2024-04-09T21:52:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo=', 'Content-Length': '0'} 2024-04-09T21:52:44.026 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021cea141023f85c3-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-09T21:52:44.027 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.027 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.027 DEBUG:boto:path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.027 DEBUG:boto:auth_path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.027 DEBUG:boto:Method: GET 2024-04-09T21:52:44.027 DEBUG:boto:Path: /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.027 DEBUG:boto:Data: 2024-04-09T21:52:44.027 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.027 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.028 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.028 DEBUG:boto:Params: {} 2024-04-09T21:52:44.028 DEBUG:boto:Token: None 2024-04-09T21:52:44.028 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.028 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-09T21:52:44.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-09T21:52:44.029 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008149bb0e1a0de1a-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.030 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.030 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.030 DEBUG:boto:path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.030 DEBUG:boto:auth_path=/xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.030 DEBUG:boto:Method: GET 2024-04-09T21:52:44.030 DEBUG:boto:Path: /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.031 DEBUG:boto:Data: 2024-04-09T21:52:44.031 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.031 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.031 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.031 DEBUG:boto:Params: {} 2024-04-09T21:52:44.031 DEBUG:boto:Token: None 2024-04-09T21:52:44.031 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46?versionId=null 2024-04-09T21:52:44.031 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-09T21:52:44.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-09T21:52:44.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011ed7d5bb5cc62fc-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.034 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.034 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.034 DEBUG:boto:path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.034 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.034 DEBUG:boto:Method: GET 2024-04-09T21:52:44.034 DEBUG:boto:Path: /xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.034 DEBUG:boto:Data: 2024-04-09T21:52:44.034 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.034 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.034 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.035 DEBUG:boto:Params: {} 2024-04-09T21:52:44.035 DEBUG:boto:Token: None 2024-04-09T21:52:44.035 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.035 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004263b1263478f1ad-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.037 DEBUG:boto:b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.037 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.037 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.038 DEBUG:boto:path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.038 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.038 DEBUG:boto:Method: GET 2024-04-09T21:52:44.038 DEBUG:boto:Path: /xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.038 DEBUG:boto:Data: 2024-04-09T21:52:44.038 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.038 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.038 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.038 DEBUG:boto:Params: {} 2024-04-09T21:52:44.038 DEBUG:boto:Token: None 2024-04-09T21:52:44.039 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.039 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dca8e84c37ba1a7-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.041 DEBUG:boto:b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.041 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.041 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.041 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.042 DEBUG:boto:Path: /xryqqk-11/obj46 2024-04-09T21:52:44.042 DEBUG:boto:Data: 2024-04-09T21:52:44.042 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.042 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.042 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.042 DEBUG:boto:Params: {} 2024-04-09T21:52:44.042 DEBUG:boto:Token: None 2024-04-09T21:52:44.042 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46 2024-04-09T21:52:44.042 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o= 2024-04-09T21:52:44.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o=', 'Content-Length': '0'} 2024-04-09T21:52:44.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f46e1732fdb59a99-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.044 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.044 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.045 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.045 DEBUG:boto:Path: /xryqqk-11/obj46 2024-04-09T21:52:44.045 DEBUG:boto:Data: 2024-04-09T21:52:44.045 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.045 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.045 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.045 DEBUG:boto:Params: {} 2024-04-09T21:52:44.045 DEBUG:boto:Token: None 2024-04-09T21:52:44.045 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46 2024-04-09T21:52:44.045 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o= 2024-04-09T21:52:44.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o=', 'Content-Length': '0'} 2024-04-09T21:52:44.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c6c55b5672e6ed5-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-09T21:52:44.047 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.047 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.047 DEBUG:boto:Method: GET 2024-04-09T21:52:44.047 DEBUG:boto:Path: /xryqqk-11/obj46 2024-04-09T21:52:44.048 DEBUG:boto:Data: 2024-04-09T21:52:44.048 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.048 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.048 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.048 DEBUG:boto:Params: {} 2024-04-09T21:52:44.048 DEBUG:boto:Token: None 2024-04-09T21:52:44.048 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46 2024-04-09T21:52:44.048 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA= 2024-04-09T21:52:44.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA=', 'Content-Length': '0'} 2024-04-09T21:52:44.050 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087cd5c16577e3732-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.050 DEBUG:boto:path=/xryqqk-11/obj46 2024-04-09T21:52:44.050 DEBUG:boto:auth_path=/xryqqk-11/obj46 2024-04-09T21:52:44.050 DEBUG:boto:Method: GET 2024-04-09T21:52:44.050 DEBUG:boto:Path: /xryqqk-11/obj46 2024-04-09T21:52:44.050 DEBUG:boto:Data: 2024-04-09T21:52:44.051 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.051 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.051 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.051 DEBUG:boto:Params: {} 2024-04-09T21:52:44.051 DEBUG:boto:Token: None 2024-04-09T21:52:44.051 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj46 2024-04-09T21:52:44.051 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA= 2024-04-09T21:52:44.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA=', 'Content-Length': '0'} 2024-04-09T21:52:44.052 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e675822cbf9824e3-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.053 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.053 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.053 DEBUG:boto:path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.053 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.053 DEBUG:boto:Method: GET 2024-04-09T21:52:44.053 DEBUG:boto:Path: /xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.053 DEBUG:boto:Data: 2024-04-09T21:52:44.054 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.054 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.054 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.054 DEBUG:boto:Params: {} 2024-04-09T21:52:44.054 DEBUG:boto:Token: None 2024-04-09T21:52:44.054 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.055 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049c567bad725bdac-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.057 DEBUG:boto:b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.057 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.057 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.057 DEBUG:boto:path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.058 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.058 DEBUG:boto:Method: GET 2024-04-09T21:52:44.058 DEBUG:boto:Path: /xryqqk-11/?prefix=obj46 2024-04-09T21:52:44.058 DEBUG:boto:Data: 2024-04-09T21:52:44.058 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.058 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.058 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.058 DEBUG:boto:Params: {} 2024-04-09T21:52:44.058 DEBUG:boto:Token: None 2024-04-09T21:52:44.058 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.059 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa7edf0634cf45ce-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.061 DEBUG:boto:b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-04-09T21:52:44.061 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.061 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.061 DEBUG:boto:path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.061 DEBUG:boto:auth_path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.062 DEBUG:boto:Method: GET 2024-04-09T21:52:44.062 DEBUG:boto:Path: /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.062 DEBUG:boto:Data: 2024-04-09T21:52:44.062 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.062 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.062 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.062 DEBUG:boto:Params: {} 2024-04-09T21:52:44.062 DEBUG:boto:Token: None 2024-04-09T21:52:44.062 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.063 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-09T21:52:44.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-09T21:52:44.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035349c4d4d3190c6-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.064 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.064 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.064 DEBUG:boto:path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.065 DEBUG:boto:auth_path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.065 DEBUG:boto:Method: GET 2024-04-09T21:52:44.065 DEBUG:boto:Path: /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.065 DEBUG:boto:Data: 2024-04-09T21:52:44.065 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.065 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.065 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.065 DEBUG:boto:Params: {} 2024-04-09T21:52:44.065 DEBUG:boto:Token: None 2024-04-09T21:52:44.065 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.066 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-09T21:52:44.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-09T21:52:44.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d08a461689d92c66-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.067 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.067 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.067 DEBUG:boto:path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.068 DEBUG:boto:auth_path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.068 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.068 DEBUG:boto:Path: /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.068 DEBUG:boto:Data: 2024-04-09T21:52:44.068 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.068 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.068 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.068 DEBUG:boto:Params: {} 2024-04-09T21:52:44.068 DEBUG:boto:Token: None 2024-04-09T21:52:44.068 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.069 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY= 2024-04-09T21:52:44.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY=', 'Content-Length': '0'} 2024-04-09T21:52:44.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069bba0a693735853-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.070 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.070 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.070 DEBUG:boto:path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.071 DEBUG:boto:auth_path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.071 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.071 DEBUG:boto:Path: /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.071 DEBUG:boto:Data: 2024-04-09T21:52:44.071 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.071 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.071 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.071 DEBUG:boto:Params: {} 2024-04-09T21:52:44.071 DEBUG:boto:Token: None 2024-04-09T21:52:44.071 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.072 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY= 2024-04-09T21:52:44.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY=', 'Content-Length': '0'} 2024-04-09T21:52:44.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000722811ce80b38e87-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-04-09T21:52:44.073 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.073 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.074 DEBUG:boto:path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.074 DEBUG:boto:auth_path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.074 DEBUG:boto:Method: GET 2024-04-09T21:52:44.074 DEBUG:boto:Path: /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.074 DEBUG:boto:Data: 2024-04-09T21:52:44.074 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.074 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.074 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.074 DEBUG:boto:Params: {} 2024-04-09T21:52:44.074 DEBUG:boto:Token: None 2024-04-09T21:52:44.075 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.075 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-09T21:52:44.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-09T21:52:44.076 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fa322ae9fea02e0-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.076 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.076 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.077 DEBUG:boto:path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.077 DEBUG:boto:auth_path=/xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.077 DEBUG:boto:Method: GET 2024-04-09T21:52:44.077 DEBUG:boto:Path: /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.077 DEBUG:boto:Data: 2024-04-09T21:52:44.077 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.077 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.077 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.077 DEBUG:boto:Params: {} 2024-04-09T21:52:44.077 DEBUG:boto:Token: None 2024-04-09T21:52:44.078 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47?versionId=null 2024-04-09T21:52:44.078 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-09T21:52:44.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-09T21:52:44.079 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073d8ed48fe22b12c-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.126 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.127 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.127 DEBUG:boto:path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.127 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.127 DEBUG:boto:Method: GET 2024-04-09T21:52:44.127 DEBUG:boto:Path: /xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.127 DEBUG:boto:Data: 2024-04-09T21:52:44.127 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.127 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.127 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.127 DEBUG:boto:Params: {} 2024-04-09T21:52:44.128 DEBUG:boto:Token: None 2024-04-09T21:52:44.128 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.128 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd968ae5f69b3bda-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.130 DEBUG:boto:b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.131 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.131 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.131 DEBUG:boto:path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.131 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.131 DEBUG:boto:Method: GET 2024-04-09T21:52:44.131 DEBUG:boto:Path: /xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.131 DEBUG:boto:Data: 2024-04-09T21:52:44.131 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.131 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.132 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.132 DEBUG:boto:Params: {} 2024-04-09T21:52:44.132 DEBUG:boto:Token: None 2024-04-09T21:52:44.132 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.132 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ada15991c199121-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.134 DEBUG:boto:b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.134 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.134 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.135 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.135 DEBUG:boto:Path: /xryqqk-11/obj47 2024-04-09T21:52:44.135 DEBUG:boto:Data: 2024-04-09T21:52:44.135 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.135 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.135 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.135 DEBUG:boto:Params: {} 2024-04-09T21:52:44.135 DEBUG:boto:Token: None 2024-04-09T21:52:44.135 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47 2024-04-09T21:52:44.135 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM= 2024-04-09T21:52:44.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM=', 'Content-Length': '0'} 2024-04-09T21:52:44.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cf05b727c46b7fd-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.137 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.137 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.137 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.137 DEBUG:boto:Path: /xryqqk-11/obj47 2024-04-09T21:52:44.137 DEBUG:boto:Data: 2024-04-09T21:52:44.138 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.138 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.138 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.138 DEBUG:boto:Params: {} 2024-04-09T21:52:44.138 DEBUG:boto:Token: None 2024-04-09T21:52:44.138 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47 2024-04-09T21:52:44.138 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM= 2024-04-09T21:52:44.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM=', 'Content-Length': '0'} 2024-04-09T21:52:44.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba00d43330d67d5e-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj47 2024-04-09T21:52:44.140 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.140 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.140 DEBUG:boto:Method: GET 2024-04-09T21:52:44.140 DEBUG:boto:Path: /xryqqk-11/obj47 2024-04-09T21:52:44.141 DEBUG:boto:Data: 2024-04-09T21:52:44.141 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.141 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.141 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.141 DEBUG:boto:Params: {} 2024-04-09T21:52:44.141 DEBUG:boto:Token: None 2024-04-09T21:52:44.141 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47 2024-04-09T21:52:44.141 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M= 2024-04-09T21:52:44.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M=', 'Content-Length': '0'} 2024-04-09T21:52:44.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b476ec3f08e1f8-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.143 DEBUG:boto:path=/xryqqk-11/obj47 2024-04-09T21:52:44.143 DEBUG:boto:auth_path=/xryqqk-11/obj47 2024-04-09T21:52:44.143 DEBUG:boto:Method: GET 2024-04-09T21:52:44.144 DEBUG:boto:Path: /xryqqk-11/obj47 2024-04-09T21:52:44.144 DEBUG:boto:Data: 2024-04-09T21:52:44.144 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.144 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.144 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.144 DEBUG:boto:Params: {} 2024-04-09T21:52:44.144 DEBUG:boto:Token: None 2024-04-09T21:52:44.144 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj47 2024-04-09T21:52:44.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M= 2024-04-09T21:52:44.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M=', 'Content-Length': '0'} 2024-04-09T21:52:44.146 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000031778d4679dae19-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.146 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.146 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.146 DEBUG:boto:path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.146 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.146 DEBUG:boto:Method: GET 2024-04-09T21:52:44.147 DEBUG:boto:Path: /xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.147 DEBUG:boto:Data: 2024-04-09T21:52:44.147 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.147 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.147 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.147 DEBUG:boto:Params: {} 2024-04-09T21:52:44.147 DEBUG:boto:Token: None 2024-04-09T21:52:44.147 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.147 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da4630c1034151c-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.150 DEBUG:boto:b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.150 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.150 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.150 DEBUG:boto:path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.150 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.150 DEBUG:boto:Method: GET 2024-04-09T21:52:44.151 DEBUG:boto:Path: /xryqqk-11/?prefix=obj47 2024-04-09T21:52:44.151 DEBUG:boto:Data: 2024-04-09T21:52:44.151 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.151 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.151 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.151 DEBUG:boto:Params: {} 2024-04-09T21:52:44.151 DEBUG:boto:Token: None 2024-04-09T21:52:44.151 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.151 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e556658300ef1e3e-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.154 DEBUG:boto:b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-04-09T21:52:44.154 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.154 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.154 DEBUG:boto:path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.154 DEBUG:boto:auth_path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.155 DEBUG:boto:Method: GET 2024-04-09T21:52:44.155 DEBUG:boto:Path: /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.155 DEBUG:boto:Data: 2024-04-09T21:52:44.155 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.155 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.155 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.155 DEBUG:boto:Params: {} 2024-04-09T21:52:44.155 DEBUG:boto:Token: None 2024-04-09T21:52:44.155 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.155 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-09T21:52:44.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-09T21:52:44.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000441f36e72a1f603e-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.157 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.158 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.158 DEBUG:boto:path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.158 DEBUG:boto:auth_path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.158 DEBUG:boto:Method: GET 2024-04-09T21:52:44.158 DEBUG:boto:Path: /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.158 DEBUG:boto:Data: 2024-04-09T21:52:44.158 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.158 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.158 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.159 DEBUG:boto:Params: {} 2024-04-09T21:52:44.159 DEBUG:boto:Token: None 2024-04-09T21:52:44.159 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.159 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-09T21:52:44.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-09T21:52:44.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002faa72a8fff2540a-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.161 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.161 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.162 DEBUG:boto:path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.162 DEBUG:boto:auth_path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.162 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.162 DEBUG:boto:Path: /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.162 DEBUG:boto:Data: 2024-04-09T21:52:44.162 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.162 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.162 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.162 DEBUG:boto:Params: {} 2024-04-09T21:52:44.162 DEBUG:boto:Token: None 2024-04-09T21:52:44.163 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.163 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg= 2024-04-09T21:52:44.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg=', 'Content-Length': '0'} 2024-04-09T21:52:44.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000488181543a9eaf3f-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.164 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.165 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.165 DEBUG:boto:path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.165 DEBUG:boto:auth_path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.165 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.165 DEBUG:boto:Path: /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.165 DEBUG:boto:Data: 2024-04-09T21:52:44.165 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.165 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.165 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.165 DEBUG:boto:Params: {} 2024-04-09T21:52:44.165 DEBUG:boto:Token: None 2024-04-09T21:52:44.166 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.166 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg= 2024-04-09T21:52:44.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg=', 'Content-Length': '0'} 2024-04-09T21:52:44.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bac3d2f231b6974-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-04-09T21:52:44.168 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.168 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.168 DEBUG:boto:path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.168 DEBUG:boto:auth_path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.168 DEBUG:boto:Method: GET 2024-04-09T21:52:44.168 DEBUG:boto:Path: /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.168 DEBUG:boto:Data: 2024-04-09T21:52:44.168 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.168 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.168 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.168 DEBUG:boto:Params: {} 2024-04-09T21:52:44.169 DEBUG:boto:Token: None 2024-04-09T21:52:44.169 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.169 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-09T21:52:44.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-09T21:52:44.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086e195f66cf4d2a6-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.171 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.171 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.171 DEBUG:boto:path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.171 DEBUG:boto:auth_path=/xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.171 DEBUG:boto:Method: GET 2024-04-09T21:52:44.171 DEBUG:boto:Path: /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.171 DEBUG:boto:Data: 2024-04-09T21:52:44.171 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.171 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.172 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.172 DEBUG:boto:Params: {} 2024-04-09T21:52:44.172 DEBUG:boto:Token: None 2024-04-09T21:52:44.172 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48?versionId=null 2024-04-09T21:52:44.172 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-09T21:52:44.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-09T21:52:44.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a019a37d663a5b3-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.174 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.174 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.174 DEBUG:boto:path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.174 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.174 DEBUG:boto:Method: GET 2024-04-09T21:52:44.174 DEBUG:boto:Path: /xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.174 DEBUG:boto:Data: 2024-04-09T21:52:44.174 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.174 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.175 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.175 DEBUG:boto:Params: {} 2024-04-09T21:52:44.175 DEBUG:boto:Token: None 2024-04-09T21:52:44.175 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.175 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008953b47e0790fed3-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.218 DEBUG:boto:b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.219 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.219 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.219 DEBUG:boto:path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.219 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.219 DEBUG:boto:Method: GET 2024-04-09T21:52:44.219 DEBUG:boto:Path: /xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.220 DEBUG:boto:Data: 2024-04-09T21:52:44.220 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.220 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.220 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.220 DEBUG:boto:Params: {} 2024-04-09T21:52:44.220 DEBUG:boto:Token: None 2024-04-09T21:52:44.220 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.220 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b5e7944215e2d8f-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.223 DEBUG:boto:b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.223 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.223 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.223 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.223 DEBUG:boto:Path: /xryqqk-11/obj48 2024-04-09T21:52:44.223 DEBUG:boto:Data: 2024-04-09T21:52:44.224 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.224 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.224 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.224 DEBUG:boto:Params: {} 2024-04-09T21:52:44.224 DEBUG:boto:Token: None 2024-04-09T21:52:44.224 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48 2024-04-09T21:52:44.224 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo= 2024-04-09T21:52:44.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo=', 'Content-Length': '0'} 2024-04-09T21:52:44.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eecd24d28128236e-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.226 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.226 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.226 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.226 DEBUG:boto:Path: /xryqqk-11/obj48 2024-04-09T21:52:44.226 DEBUG:boto:Data: 2024-04-09T21:52:44.226 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.227 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.227 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.227 DEBUG:boto:Params: {} 2024-04-09T21:52:44.227 DEBUG:boto:Token: None 2024-04-09T21:52:44.227 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48 2024-04-09T21:52:44.227 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo= 2024-04-09T21:52:44.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo=', 'Content-Length': '0'} 2024-04-09T21:52:44.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b53542bf228fcf45-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj48 2024-04-09T21:52:44.229 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.229 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.229 DEBUG:boto:Method: GET 2024-04-09T21:52:44.229 DEBUG:boto:Path: /xryqqk-11/obj48 2024-04-09T21:52:44.229 DEBUG:boto:Data: 2024-04-09T21:52:44.229 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.229 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.229 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.229 DEBUG:boto:Params: {} 2024-04-09T21:52:44.230 DEBUG:boto:Token: None 2024-04-09T21:52:44.230 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48 2024-04-09T21:52:44.230 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw= 2024-04-09T21:52:44.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw=', 'Content-Length': '0'} 2024-04-09T21:52:44.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e099ce3720ff796a-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.231 DEBUG:boto:path=/xryqqk-11/obj48 2024-04-09T21:52:44.232 DEBUG:boto:auth_path=/xryqqk-11/obj48 2024-04-09T21:52:44.232 DEBUG:boto:Method: GET 2024-04-09T21:52:44.232 DEBUG:boto:Path: /xryqqk-11/obj48 2024-04-09T21:52:44.232 DEBUG:boto:Data: 2024-04-09T21:52:44.232 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.232 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.232 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.232 DEBUG:boto:Params: {} 2024-04-09T21:52:44.232 DEBUG:boto:Token: None 2024-04-09T21:52:44.232 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj48 2024-04-09T21:52:44.233 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw= 2024-04-09T21:52:44.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw=', 'Content-Length': '0'} 2024-04-09T21:52:44.234 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6eef1a686bcc60c-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.234 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.234 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.235 DEBUG:boto:path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.235 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.235 DEBUG:boto:Method: GET 2024-04-09T21:52:44.235 DEBUG:boto:Path: /xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.235 DEBUG:boto:Data: 2024-04-09T21:52:44.235 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.235 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.235 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.235 DEBUG:boto:Params: {} 2024-04-09T21:52:44.235 DEBUG:boto:Token: None 2024-04-09T21:52:44.235 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.236 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676da375805812cc-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.238 DEBUG:boto:b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.239 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.239 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.239 DEBUG:boto:path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.239 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.239 DEBUG:boto:Method: GET 2024-04-09T21:52:44.239 DEBUG:boto:Path: /xryqqk-11/?prefix=obj48 2024-04-09T21:52:44.239 DEBUG:boto:Data: 2024-04-09T21:52:44.239 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.239 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.239 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.239 DEBUG:boto:Params: {} 2024-04-09T21:52:44.240 DEBUG:boto:Token: None 2024-04-09T21:52:44.240 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb229b6eed8e1bf0-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.242 DEBUG:boto:b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-04-09T21:52:44.242 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.243 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.243 DEBUG:boto:path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.243 DEBUG:boto:auth_path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.243 DEBUG:boto:Method: GET 2024-04-09T21:52:44.243 DEBUG:boto:Path: /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.243 DEBUG:boto:Data: 2024-04-09T21:52:44.243 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.243 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.243 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.243 DEBUG:boto:Params: {} 2024-04-09T21:52:44.244 DEBUG:boto:Token: None 2024-04-09T21:52:44.244 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.244 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-09T21:52:44.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-09T21:52:44.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000955d05aa46cc4789-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.245 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.246 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.246 DEBUG:boto:path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.246 DEBUG:boto:auth_path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.246 DEBUG:boto:Method: GET 2024-04-09T21:52:44.246 DEBUG:boto:Path: /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.246 DEBUG:boto:Data: 2024-04-09T21:52:44.246 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.246 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.246 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.246 DEBUG:boto:Params: {} 2024-04-09T21:52:44.247 DEBUG:boto:Token: None 2024-04-09T21:52:44.247 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.247 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-09T21:52:44.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-09T21:52:44.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031eb36bacb00398e-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.249 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.249 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.249 DEBUG:boto:path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.249 DEBUG:boto:auth_path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.249 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.249 DEBUG:boto:Path: /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.249 DEBUG:boto:Data: 2024-04-09T21:52:44.249 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.249 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.249 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.250 DEBUG:boto:Params: {} 2024-04-09T21:52:44.250 DEBUG:boto:Token: None 2024-04-09T21:52:44.250 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.250 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U= 2024-04-09T21:52:44.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U=', 'Content-Length': '0'} 2024-04-09T21:52:44.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028af6574839e9e34-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.252 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.252 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.252 DEBUG:boto:path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.252 DEBUG:boto:auth_path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.252 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.252 DEBUG:boto:Path: /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.252 DEBUG:boto:Data: 2024-04-09T21:52:44.252 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.252 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.252 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.252 DEBUG:boto:Params: {} 2024-04-09T21:52:44.253 DEBUG:boto:Token: None 2024-04-09T21:52:44.253 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.253 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U= 2024-04-09T21:52:44.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U=', 'Content-Length': '0'} 2024-04-09T21:52:44.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001af6ca1814c151e4-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-04-09T21:52:44.255 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.255 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.255 DEBUG:boto:path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.255 DEBUG:boto:auth_path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.255 DEBUG:boto:Method: GET 2024-04-09T21:52:44.255 DEBUG:boto:Path: /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.255 DEBUG:boto:Data: 2024-04-09T21:52:44.255 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.255 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.255 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.255 DEBUG:boto:Params: {} 2024-04-09T21:52:44.256 DEBUG:boto:Token: None 2024-04-09T21:52:44.256 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.256 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-09T21:52:44.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-09T21:52:44.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707d47179f994eeb-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.258 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.258 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.258 DEBUG:boto:path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.258 DEBUG:boto:auth_path=/xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.258 DEBUG:boto:Method: GET 2024-04-09T21:52:44.258 DEBUG:boto:Path: /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.258 DEBUG:boto:Data: 2024-04-09T21:52:44.258 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.258 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.259 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.259 DEBUG:boto:Params: {} 2024-04-09T21:52:44.259 DEBUG:boto:Token: None 2024-04-09T21:52:44.259 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49?versionId=null 2024-04-09T21:52:44.259 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-09T21:52:44.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-09T21:52:44.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc15cc6425ec406-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.261 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.261 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.261 DEBUG:boto:path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.261 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.261 DEBUG:boto:Method: GET 2024-04-09T21:52:44.261 DEBUG:boto:Path: /xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.261 DEBUG:boto:Data: 2024-04-09T21:52:44.261 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.261 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.261 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.262 DEBUG:boto:Params: {} 2024-04-09T21:52:44.262 DEBUG:boto:Token: None 2024-04-09T21:52:44.262 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.262 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087ac07b43fadee6-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.264 DEBUG:boto:b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.264 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.265 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.265 DEBUG:boto:path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.265 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.265 DEBUG:boto:Method: GET 2024-04-09T21:52:44.265 DEBUG:boto:Path: /xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.265 DEBUG:boto:Data: 2024-04-09T21:52:44.265 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.265 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.265 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.265 DEBUG:boto:Params: {} 2024-04-09T21:52:44.265 DEBUG:boto:Token: None 2024-04-09T21:52:44.266 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.266 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.268 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005919b22267968776-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.268 DEBUG:boto:b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.268 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.268 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.268 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.269 DEBUG:boto:Path: /xryqqk-11/obj49 2024-04-09T21:52:44.269 DEBUG:boto:Data: 2024-04-09T21:52:44.269 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.269 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.269 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.269 DEBUG:boto:Params: {} 2024-04-09T21:52:44.269 DEBUG:boto:Token: None 2024-04-09T21:52:44.269 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49 2024-04-09T21:52:44.269 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw= 2024-04-09T21:52:44.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw=', 'Content-Length': '0'} 2024-04-09T21:52:44.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095ffbaa53870867-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.271 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.271 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.272 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.272 DEBUG:boto:Path: /xryqqk-11/obj49 2024-04-09T21:52:44.272 DEBUG:boto:Data: 2024-04-09T21:52:44.272 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.272 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.272 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.272 DEBUG:boto:Params: {} 2024-04-09T21:52:44.272 DEBUG:boto:Token: None 2024-04-09T21:52:44.272 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49 2024-04-09T21:52:44.272 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw= 2024-04-09T21:52:44.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw=', 'Content-Length': '0'} 2024-04-09T21:52:44.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a06d0207a78f1e69-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj49 2024-04-09T21:52:44.275 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.275 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.275 DEBUG:boto:Method: GET 2024-04-09T21:52:44.276 DEBUG:boto:Path: /xryqqk-11/obj49 2024-04-09T21:52:44.276 DEBUG:boto:Data: 2024-04-09T21:52:44.276 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.277 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.277 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.277 DEBUG:boto:Params: {} 2024-04-09T21:52:44.278 DEBUG:boto:Token: None 2024-04-09T21:52:44.278 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49 2024-04-09T21:52:44.278 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c= 2024-04-09T21:52:44.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c=', 'Content-Length': '0'} 2024-04-09T21:52:44.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8fe037875367546-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.281 DEBUG:boto:path=/xryqqk-11/obj49 2024-04-09T21:52:44.281 DEBUG:boto:auth_path=/xryqqk-11/obj49 2024-04-09T21:52:44.282 DEBUG:boto:Method: GET 2024-04-09T21:52:44.282 DEBUG:boto:Path: /xryqqk-11/obj49 2024-04-09T21:52:44.282 DEBUG:boto:Data: 2024-04-09T21:52:44.283 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.283 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.284 DEBUG:boto:Params: {} 2024-04-09T21:52:44.284 DEBUG:boto:Token: None 2024-04-09T21:52:44.284 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj49 2024-04-09T21:52:44.285 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c= 2024-04-09T21:52:44.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c=', 'Content-Length': '0'} 2024-04-09T21:52:44.286 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f6d20930d385e07-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.287 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.287 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.287 DEBUG:boto:path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.287 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.287 DEBUG:boto:Method: GET 2024-04-09T21:52:44.288 DEBUG:boto:Path: /xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.288 DEBUG:boto:Data: 2024-04-09T21:52:44.288 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.288 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.288 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.288 DEBUG:boto:Params: {} 2024-04-09T21:52:44.288 DEBUG:boto:Token: None 2024-04-09T21:52:44.288 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.288 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0556c030efa7a1c-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.291 DEBUG:boto:b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.291 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.291 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.291 DEBUG:boto:path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.292 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.292 DEBUG:boto:Method: GET 2024-04-09T21:52:44.292 DEBUG:boto:Path: /xryqqk-11/?prefix=obj49 2024-04-09T21:52:44.292 DEBUG:boto:Data: 2024-04-09T21:52:44.292 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.292 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.292 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.292 DEBUG:boto:Params: {} 2024-04-09T21:52:44.292 DEBUG:boto:Token: None 2024-04-09T21:52:44.293 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.293 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fae432474d9062-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.338 DEBUG:boto:b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-09T21:52:44.339 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.339 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.339 DEBUG:boto:path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.339 DEBUG:boto:auth_path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.339 DEBUG:boto:Method: GET 2024-04-09T21:52:44.339 DEBUG:boto:Path: /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.340 DEBUG:boto:Data: 2024-04-09T21:52:44.340 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.340 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.340 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.340 DEBUG:boto:Params: {} 2024-04-09T21:52:44.340 DEBUG:boto:Token: None 2024-04-09T21:52:44.340 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.340 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-09T21:52:44.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-09T21:52:44.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047e9228716106061-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.342 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.343 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.343 DEBUG:boto:path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.343 DEBUG:boto:auth_path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.343 DEBUG:boto:Method: GET 2024-04-09T21:52:44.343 DEBUG:boto:Path: /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.343 DEBUG:boto:Data: 2024-04-09T21:52:44.343 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.343 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.343 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.343 DEBUG:boto:Params: {} 2024-04-09T21:52:44.344 DEBUG:boto:Token: None 2024-04-09T21:52:44.344 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.344 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-09T21:52:44.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-09T21:52:44.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000451e63f707f197e4-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.346 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.346 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.346 DEBUG:boto:path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.346 DEBUG:boto:auth_path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.346 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.346 DEBUG:boto:Path: /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.346 DEBUG:boto:Data: 2024-04-09T21:52:44.346 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.346 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.347 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.347 DEBUG:boto:Params: {} 2024-04-09T21:52:44.347 DEBUG:boto:Token: None 2024-04-09T21:52:44.347 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.347 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ= 2024-04-09T21:52:44.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ=', 'Content-Length': '0'} 2024-04-09T21:52:44.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000087e0cec750d0c39-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.349 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.350 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.350 DEBUG:boto:path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.350 DEBUG:boto:auth_path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.350 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.350 DEBUG:boto:Path: /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.350 DEBUG:boto:Data: 2024-04-09T21:52:44.350 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.350 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.350 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.350 DEBUG:boto:Params: {} 2024-04-09T21:52:44.350 DEBUG:boto:Token: None 2024-04-09T21:52:44.351 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.351 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ= 2024-04-09T21:52:44.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ=', 'Content-Length': '0'} 2024-04-09T21:52:44.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009662b7c87c971f41-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-09T21:52:44.352 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.353 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.353 DEBUG:boto:path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.353 DEBUG:boto:auth_path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.353 DEBUG:boto:Method: GET 2024-04-09T21:52:44.353 DEBUG:boto:Path: /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.353 DEBUG:boto:Data: 2024-04-09T21:52:44.353 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.353 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.353 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.353 DEBUG:boto:Params: {} 2024-04-09T21:52:44.353 DEBUG:boto:Token: None 2024-04-09T21:52:44.354 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.354 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-09T21:52:44.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-09T21:52:44.355 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a30b1dcbd17873c0-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.402 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.403 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.403 DEBUG:boto:path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.403 DEBUG:boto:auth_path=/xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.403 DEBUG:boto:Method: GET 2024-04-09T21:52:44.403 DEBUG:boto:Path: /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.403 DEBUG:boto:Data: 2024-04-09T21:52:44.403 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.403 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.403 DEBUG:boto:Params: {} 2024-04-09T21:52:44.404 DEBUG:boto:Token: None 2024-04-09T21:52:44.404 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5?versionId=null 2024-04-09T21:52:44.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-09T21:52:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-09T21:52:44.406 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fbf782e47da43cf-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.406 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.406 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.407 DEBUG:boto:path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.407 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.407 DEBUG:boto:Method: GET 2024-04-09T21:52:44.407 DEBUG:boto:Path: /xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.407 DEBUG:boto:Data: 2024-04-09T21:52:44.407 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.407 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.407 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.407 DEBUG:boto:Params: {} 2024-04-09T21:52:44.407 DEBUG:boto:Token: None 2024-04-09T21:52:44.408 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.408 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162254f3f765c2e2-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.410 DEBUG:boto:b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.410 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.410 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.411 DEBUG:boto:path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.411 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.411 DEBUG:boto:Method: GET 2024-04-09T21:52:44.411 DEBUG:boto:Path: /xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.411 DEBUG:boto:Data: 2024-04-09T21:52:44.411 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.411 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.411 DEBUG:boto:Params: {} 2024-04-09T21:52:44.411 DEBUG:boto:Token: None 2024-04-09T21:52:44.412 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.412 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a468290ed320ff84-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.414 DEBUG:boto:b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.414 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.415 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.415 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.415 DEBUG:boto:Path: /xryqqk-11/obj5 2024-04-09T21:52:44.415 DEBUG:boto:Data: 2024-04-09T21:52:44.415 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.415 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.415 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.415 DEBUG:boto:Params: {} 2024-04-09T21:52:44.415 DEBUG:boto:Token: None 2024-04-09T21:52:44.416 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5 2024-04-09T21:52:44.416 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg= 2024-04-09T21:52:44.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg=', 'Content-Length': '0'} 2024-04-09T21:52:44.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbf0f44b20ac0e4b-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.417 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.417 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.418 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.418 DEBUG:boto:Path: /xryqqk-11/obj5 2024-04-09T21:52:44.418 DEBUG:boto:Data: 2024-04-09T21:52:44.418 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.418 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.418 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.418 DEBUG:boto:Params: {} 2024-04-09T21:52:44.418 DEBUG:boto:Token: None 2024-04-09T21:52:44.418 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5 2024-04-09T21:52:44.418 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg= 2024-04-09T21:52:44.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg=', 'Content-Length': '0'} 2024-04-09T21:52:44.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000049ddfbf9f21f23d-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-09T21:52:44.420 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.420 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.420 DEBUG:boto:Method: GET 2024-04-09T21:52:44.421 DEBUG:boto:Path: /xryqqk-11/obj5 2024-04-09T21:52:44.421 DEBUG:boto:Data: 2024-04-09T21:52:44.421 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.421 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.421 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.421 DEBUG:boto:Params: {} 2024-04-09T21:52:44.421 DEBUG:boto:Token: None 2024-04-09T21:52:44.421 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5 2024-04-09T21:52:44.421 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo= 2024-04-09T21:52:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo=', 'Content-Length': '0'} 2024-04-09T21:52:44.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b72f7b573fda36e2-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.423 DEBUG:boto:path=/xryqqk-11/obj5 2024-04-09T21:52:44.423 DEBUG:boto:auth_path=/xryqqk-11/obj5 2024-04-09T21:52:44.423 DEBUG:boto:Method: GET 2024-04-09T21:52:44.423 DEBUG:boto:Path: /xryqqk-11/obj5 2024-04-09T21:52:44.423 DEBUG:boto:Data: 2024-04-09T21:52:44.424 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.424 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.424 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.424 DEBUG:boto:Params: {} 2024-04-09T21:52:44.424 DEBUG:boto:Token: None 2024-04-09T21:52:44.424 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj5 2024-04-09T21:52:44.424 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo= 2024-04-09T21:52:44.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo=', 'Content-Length': '0'} 2024-04-09T21:52:44.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad355b9e5f7601e-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.426 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.426 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.426 DEBUG:boto:path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.426 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.427 DEBUG:boto:Method: GET 2024-04-09T21:52:44.427 DEBUG:boto:Path: /xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.427 DEBUG:boto:Data: 2024-04-09T21:52:44.427 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.427 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.427 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.427 DEBUG:boto:Params: {} 2024-04-09T21:52:44.427 DEBUG:boto:Token: None 2024-04-09T21:52:44.427 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.427 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e84cd9967c4812-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.430 DEBUG:boto:b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.430 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.430 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.430 DEBUG:boto:path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.430 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.430 DEBUG:boto:Method: GET 2024-04-09T21:52:44.430 DEBUG:boto:Path: /xryqqk-11/?prefix=obj5 2024-04-09T21:52:44.431 DEBUG:boto:Data: 2024-04-09T21:52:44.431 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.431 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.431 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.431 DEBUG:boto:Params: {} 2024-04-09T21:52:44.431 DEBUG:boto:Token: None 2024-04-09T21:52:44.431 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.431 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96c22998ef9ebf7-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.433 DEBUG:boto:b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-09T21:52:44.434 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.434 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.434 DEBUG:boto:path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.434 DEBUG:boto:auth_path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.434 DEBUG:boto:Method: GET 2024-04-09T21:52:44.434 DEBUG:boto:Path: /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.434 DEBUG:boto:Data: 2024-04-09T21:52:44.434 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.435 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.435 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.435 DEBUG:boto:Params: {} 2024-04-09T21:52:44.435 DEBUG:boto:Token: None 2024-04-09T21:52:44.435 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.435 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-09T21:52:44.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-09T21:52:44.437 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003477893755c5c9a8-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.437 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.437 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.437 DEBUG:boto:path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.438 DEBUG:boto:auth_path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.438 DEBUG:boto:Method: GET 2024-04-09T21:52:44.438 DEBUG:boto:Path: /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.438 DEBUG:boto:Data: 2024-04-09T21:52:44.438 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.438 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.438 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.438 DEBUG:boto:Params: {} 2024-04-09T21:52:44.438 DEBUG:boto:Token: None 2024-04-09T21:52:44.438 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.439 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-09T21:52:44.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-09T21:52:44.440 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a194e85ce7f5332a-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.440 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.440 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.440 DEBUG:boto:path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.441 DEBUG:boto:auth_path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.441 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.441 DEBUG:boto:Path: /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.441 DEBUG:boto:Data: 2024-04-09T21:52:44.441 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.441 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.441 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.441 DEBUG:boto:Params: {} 2024-04-09T21:52:44.441 DEBUG:boto:Token: None 2024-04-09T21:52:44.441 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.442 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo= 2024-04-09T21:52:44.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo=', 'Content-Length': '0'} 2024-04-09T21:52:44.443 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c75ca16a87638544-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.443 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.443 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.444 DEBUG:boto:path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.444 DEBUG:boto:auth_path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.444 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.444 DEBUG:boto:Path: /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.444 DEBUG:boto:Data: 2024-04-09T21:52:44.444 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.444 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.444 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.444 DEBUG:boto:Params: {} 2024-04-09T21:52:44.444 DEBUG:boto:Token: None 2024-04-09T21:52:44.444 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.445 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo= 2024-04-09T21:52:44.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo=', 'Content-Length': '0'} 2024-04-09T21:52:44.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb2e51f6cf57e396-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-09T21:52:44.447 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.447 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.447 DEBUG:boto:path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.447 DEBUG:boto:auth_path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.447 DEBUG:boto:Method: GET 2024-04-09T21:52:44.447 DEBUG:boto:Path: /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.447 DEBUG:boto:Data: 2024-04-09T21:52:44.447 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.447 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.447 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.447 DEBUG:boto:Params: {} 2024-04-09T21:52:44.448 DEBUG:boto:Token: None 2024-04-09T21:52:44.448 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.448 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-09T21:52:44.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-09T21:52:44.449 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7e2838190c91855-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.449 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.450 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.450 DEBUG:boto:path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.450 DEBUG:boto:auth_path=/xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.450 DEBUG:boto:Method: GET 2024-04-09T21:52:44.450 DEBUG:boto:Path: /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.450 DEBUG:boto:Data: 2024-04-09T21:52:44.450 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.450 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.450 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.450 DEBUG:boto:Params: {} 2024-04-09T21:52:44.451 DEBUG:boto:Token: None 2024-04-09T21:52:44.451 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6?versionId=null 2024-04-09T21:52:44.451 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-09T21:52:44.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-09T21:52:44.452 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097acaac7334fdb4a-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.452 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.453 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.453 DEBUG:boto:path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.453 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.453 DEBUG:boto:Method: GET 2024-04-09T21:52:44.453 DEBUG:boto:Path: /xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.453 DEBUG:boto:Data: 2024-04-09T21:52:44.453 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.453 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.453 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.453 DEBUG:boto:Params: {} 2024-04-09T21:52:44.454 DEBUG:boto:Token: None 2024-04-09T21:52:44.454 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.454 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8c3c76a17e37a9-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.456 DEBUG:boto:b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.456 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.457 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.457 DEBUG:boto:path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.457 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.457 DEBUG:boto:Method: GET 2024-04-09T21:52:44.457 DEBUG:boto:Path: /xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.457 DEBUG:boto:Data: 2024-04-09T21:52:44.457 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.457 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.457 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.457 DEBUG:boto:Params: {} 2024-04-09T21:52:44.457 DEBUG:boto:Token: None 2024-04-09T21:52:44.457 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.458 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3a9db85baac754-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.460 DEBUG:boto:b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.460 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.460 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.460 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.460 DEBUG:boto:Path: /xryqqk-11/obj6 2024-04-09T21:52:44.460 DEBUG:boto:Data: 2024-04-09T21:52:44.461 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.461 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.461 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.461 DEBUG:boto:Params: {} 2024-04-09T21:52:44.461 DEBUG:boto:Token: None 2024-04-09T21:52:44.461 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6 2024-04-09T21:52:44.461 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8= 2024-04-09T21:52:44.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8=', 'Content-Length': '0'} 2024-04-09T21:52:44.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dda5b03fdb20d7c-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.463 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.463 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.464 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.464 DEBUG:boto:Path: /xryqqk-11/obj6 2024-04-09T21:52:44.464 DEBUG:boto:Data: 2024-04-09T21:52:44.464 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.464 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.464 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.464 DEBUG:boto:Params: {} 2024-04-09T21:52:44.464 DEBUG:boto:Token: None 2024-04-09T21:52:44.464 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6 2024-04-09T21:52:44.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8= 2024-04-09T21:52:44.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8=', 'Content-Length': '0'} 2024-04-09T21:52:44.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adce83c7eb242f48-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-09T21:52:44.466 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.466 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.466 DEBUG:boto:Method: GET 2024-04-09T21:52:44.466 DEBUG:boto:Path: /xryqqk-11/obj6 2024-04-09T21:52:44.467 DEBUG:boto:Data: 2024-04-09T21:52:44.467 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.467 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.467 DEBUG:boto:Params: {} 2024-04-09T21:52:44.467 DEBUG:boto:Token: None 2024-04-09T21:52:44.467 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6 2024-04-09T21:52:44.467 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4= 2024-04-09T21:52:44.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4=', 'Content-Length': '0'} 2024-04-09T21:52:44.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c3cc076e00e3eaa-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.514 DEBUG:boto:path=/xryqqk-11/obj6 2024-04-09T21:52:44.515 DEBUG:boto:auth_path=/xryqqk-11/obj6 2024-04-09T21:52:44.515 DEBUG:boto:Method: GET 2024-04-09T21:52:44.515 DEBUG:boto:Path: /xryqqk-11/obj6 2024-04-09T21:52:44.515 DEBUG:boto:Data: 2024-04-09T21:52:44.515 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.515 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.515 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.516 DEBUG:boto:Params: {} 2024-04-09T21:52:44.517 DEBUG:boto:Token: None 2024-04-09T21:52:44.517 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj6 2024-04-09T21:52:44.517 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4= 2024-04-09T21:52:44.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4=', 'Content-Length': '0'} 2024-04-09T21:52:44.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfe7d681bbcbabad-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.519 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.519 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.519 DEBUG:boto:path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.520 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.520 DEBUG:boto:Method: GET 2024-04-09T21:52:44.520 DEBUG:boto:Path: /xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.520 DEBUG:boto:Data: 2024-04-09T21:52:44.520 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.520 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.520 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.520 DEBUG:boto:Params: {} 2024-04-09T21:52:44.520 DEBUG:boto:Token: None 2024-04-09T21:52:44.521 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.521 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008304f812ea637ec7-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.523 DEBUG:boto:b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.524 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.524 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.524 DEBUG:boto:path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.524 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.524 DEBUG:boto:Method: GET 2024-04-09T21:52:44.524 DEBUG:boto:Path: /xryqqk-11/?prefix=obj6 2024-04-09T21:52:44.524 DEBUG:boto:Data: 2024-04-09T21:52:44.524 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.525 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.525 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.525 DEBUG:boto:Params: {} 2024-04-09T21:52:44.525 DEBUG:boto:Token: None 2024-04-09T21:52:44.525 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.525 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d136c58100e98160-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.529 DEBUG:boto:b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-09T21:52:44.530 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.530 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.530 DEBUG:boto:path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.530 DEBUG:boto:auth_path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.531 DEBUG:boto:Method: GET 2024-04-09T21:52:44.531 DEBUG:boto:Path: /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.531 DEBUG:boto:Data: 2024-04-09T21:52:44.531 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.531 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.531 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.531 DEBUG:boto:Params: {} 2024-04-09T21:52:44.531 DEBUG:boto:Token: None 2024-04-09T21:52:44.532 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-09T21:52:44.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-09T21:52:44.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014c1dad13302cc00-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.534 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.534 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.534 DEBUG:boto:path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.534 DEBUG:boto:auth_path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.535 DEBUG:boto:Method: GET 2024-04-09T21:52:44.535 DEBUG:boto:Path: /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.535 DEBUG:boto:Data: 2024-04-09T21:52:44.535 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.535 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.535 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.535 DEBUG:boto:Params: {} 2024-04-09T21:52:44.535 DEBUG:boto:Token: None 2024-04-09T21:52:44.535 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-09T21:52:44.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-09T21:52:44.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e52f1adb5c2ffa-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.578 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.579 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.579 DEBUG:boto:path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.579 DEBUG:boto:auth_path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.579 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.579 DEBUG:boto:Path: /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.579 DEBUG:boto:Data: 2024-04-09T21:52:44.579 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.579 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.579 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.579 DEBUG:boto:Params: {} 2024-04-09T21:52:44.580 DEBUG:boto:Token: None 2024-04-09T21:52:44.580 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.580 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8= 2024-04-09T21:52:44.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8=', 'Content-Length': '0'} 2024-04-09T21:52:44.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018bdff3660850448-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.582 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.582 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.582 DEBUG:boto:path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.582 DEBUG:boto:auth_path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.582 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.582 DEBUG:boto:Path: /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.582 DEBUG:boto:Data: 2024-04-09T21:52:44.582 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.582 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.582 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.583 DEBUG:boto:Params: {} 2024-04-09T21:52:44.583 DEBUG:boto:Token: None 2024-04-09T21:52:44.583 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.583 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8= 2024-04-09T21:52:44.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8=', 'Content-Length': '0'} 2024-04-09T21:52:44.584 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077467ffe8de1fda7-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-09T21:52:44.585 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.585 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.585 DEBUG:boto:path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.585 DEBUG:boto:auth_path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.585 DEBUG:boto:Method: GET 2024-04-09T21:52:44.585 DEBUG:boto:Path: /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.585 DEBUG:boto:Data: 2024-04-09T21:52:44.585 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.585 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.586 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.586 DEBUG:boto:Params: {} 2024-04-09T21:52:44.586 DEBUG:boto:Token: None 2024-04-09T21:52:44.586 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.586 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-09T21:52:44.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-09T21:52:44.588 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e7f79e82c3ba80d-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.588 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.588 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.588 DEBUG:boto:path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.588 DEBUG:boto:auth_path=/xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.588 DEBUG:boto:Method: GET 2024-04-09T21:52:44.588 DEBUG:boto:Path: /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.589 DEBUG:boto:Data: 2024-04-09T21:52:44.589 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.589 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.589 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.589 DEBUG:boto:Params: {} 2024-04-09T21:52:44.589 DEBUG:boto:Token: None 2024-04-09T21:52:44.589 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7?versionId=null 2024-04-09T21:52:44.589 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-09T21:52:44.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-09T21:52:44.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb9e734ab0d036da-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.591 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.591 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.591 DEBUG:boto:path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.591 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.592 DEBUG:boto:Method: GET 2024-04-09T21:52:44.592 DEBUG:boto:Path: /xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.592 DEBUG:boto:Data: 2024-04-09T21:52:44.592 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.592 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.592 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.592 DEBUG:boto:Params: {} 2024-04-09T21:52:44.592 DEBUG:boto:Token: None 2024-04-09T21:52:44.592 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.593 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312f6c78fc37fbb9-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.595 DEBUG:boto:b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.595 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.595 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.596 DEBUG:boto:path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.596 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.596 DEBUG:boto:Method: GET 2024-04-09T21:52:44.596 DEBUG:boto:Path: /xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.596 DEBUG:boto:Data: 2024-04-09T21:52:44.596 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.596 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.596 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.596 DEBUG:boto:Params: {} 2024-04-09T21:52:44.596 DEBUG:boto:Token: None 2024-04-09T21:52:44.597 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.597 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359d250969614b89-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.599 DEBUG:boto:b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.599 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.600 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.600 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.600 DEBUG:boto:Path: /xryqqk-11/obj7 2024-04-09T21:52:44.600 DEBUG:boto:Data: 2024-04-09T21:52:44.600 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.600 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.600 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.600 DEBUG:boto:Params: {} 2024-04-09T21:52:44.600 DEBUG:boto:Token: None 2024-04-09T21:52:44.600 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7 2024-04-09T21:52:44.601 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ= 2024-04-09T21:52:44.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ=', 'Content-Length': '0'} 2024-04-09T21:52:44.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a1807f717067148-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.602 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.603 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.603 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.603 DEBUG:boto:Path: /xryqqk-11/obj7 2024-04-09T21:52:44.603 DEBUG:boto:Data: 2024-04-09T21:52:44.603 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.603 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.603 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.603 DEBUG:boto:Params: {} 2024-04-09T21:52:44.603 DEBUG:boto:Token: None 2024-04-09T21:52:44.603 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7 2024-04-09T21:52:44.604 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ= 2024-04-09T21:52:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ=', 'Content-Length': '0'} 2024-04-09T21:52:44.605 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c3884f2145feaca-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-09T21:52:44.605 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.605 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.606 DEBUG:boto:Method: GET 2024-04-09T21:52:44.606 DEBUG:boto:Path: /xryqqk-11/obj7 2024-04-09T21:52:44.606 DEBUG:boto:Data: 2024-04-09T21:52:44.606 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.606 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.606 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.606 DEBUG:boto:Params: {} 2024-04-09T21:52:44.606 DEBUG:boto:Token: None 2024-04-09T21:52:44.606 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7 2024-04-09T21:52:44.606 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY= 2024-04-09T21:52:44.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY=', 'Content-Length': '0'} 2024-04-09T21:52:44.608 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab0075c2aa674e50-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.650 DEBUG:boto:path=/xryqqk-11/obj7 2024-04-09T21:52:44.651 DEBUG:boto:auth_path=/xryqqk-11/obj7 2024-04-09T21:52:44.651 DEBUG:boto:Method: GET 2024-04-09T21:52:44.651 DEBUG:boto:Path: /xryqqk-11/obj7 2024-04-09T21:52:44.651 DEBUG:boto:Data: 2024-04-09T21:52:44.651 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.651 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.651 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.651 DEBUG:boto:Params: {} 2024-04-09T21:52:44.652 DEBUG:boto:Token: None 2024-04-09T21:52:44.652 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj7 2024-04-09T21:52:44.652 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY= 2024-04-09T21:52:44.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY=', 'Content-Length': '0'} 2024-04-09T21:52:44.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000067674d77c085512-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.654 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.654 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.654 DEBUG:boto:path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.654 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.654 DEBUG:boto:Method: GET 2024-04-09T21:52:44.654 DEBUG:boto:Path: /xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.655 DEBUG:boto:Data: 2024-04-09T21:52:44.655 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.655 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.655 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.655 DEBUG:boto:Params: {} 2024-04-09T21:52:44.655 DEBUG:boto:Token: None 2024-04-09T21:52:44.655 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.655 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007714382d4499480a-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.658 DEBUG:boto:b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.658 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.658 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.659 DEBUG:boto:path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.659 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.659 DEBUG:boto:Method: GET 2024-04-09T21:52:44.659 DEBUG:boto:Path: /xryqqk-11/?prefix=obj7 2024-04-09T21:52:44.659 DEBUG:boto:Data: 2024-04-09T21:52:44.659 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.659 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.659 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.659 DEBUG:boto:Params: {} 2024-04-09T21:52:44.660 DEBUG:boto:Token: None 2024-04-09T21:52:44.660 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.660 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003648e11dfc4427a6-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.662 DEBUG:boto:b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-09T21:52:44.663 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.663 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.663 DEBUG:boto:path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.663 DEBUG:boto:auth_path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.663 DEBUG:boto:Method: GET 2024-04-09T21:52:44.663 DEBUG:boto:Path: /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.664 DEBUG:boto:Data: 2024-04-09T21:52:44.664 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.664 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.664 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.664 DEBUG:boto:Params: {} 2024-04-09T21:52:44.664 DEBUG:boto:Token: None 2024-04-09T21:52:44.664 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-09T21:52:44.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-09T21:52:44.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095096bb2e6d81f28-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.666 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.667 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.667 DEBUG:boto:path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.667 DEBUG:boto:auth_path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.667 DEBUG:boto:Method: GET 2024-04-09T21:52:44.667 DEBUG:boto:Path: /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.667 DEBUG:boto:Data: 2024-04-09T21:52:44.667 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.667 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.667 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.667 DEBUG:boto:Params: {} 2024-04-09T21:52:44.668 DEBUG:boto:Token: None 2024-04-09T21:52:44.668 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.668 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-09T21:52:44.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-09T21:52:44.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000061f6376d4f8abae-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.670 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.670 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.670 DEBUG:boto:path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.670 DEBUG:boto:auth_path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.671 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.671 DEBUG:boto:Path: /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.671 DEBUG:boto:Data: 2024-04-09T21:52:44.671 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.671 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.671 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.671 DEBUG:boto:Params: {} 2024-04-09T21:52:44.671 DEBUG:boto:Token: None 2024-04-09T21:52:44.671 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.671 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA= 2024-04-09T21:52:44.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA=', 'Content-Length': '0'} 2024-04-09T21:52:44.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cea0c98201b137c-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.674 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.674 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.674 DEBUG:boto:path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.674 DEBUG:boto:auth_path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.674 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.674 DEBUG:boto:Path: /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.674 DEBUG:boto:Data: 2024-04-09T21:52:44.674 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.674 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.675 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.675 DEBUG:boto:Params: {} 2024-04-09T21:52:44.675 DEBUG:boto:Token: None 2024-04-09T21:52:44.675 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.675 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA= 2024-04-09T21:52:44.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA=', 'Content-Length': '0'} 2024-04-09T21:52:44.676 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b15279e676555cf-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-09T21:52:44.677 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.677 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.677 DEBUG:boto:path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.677 DEBUG:boto:auth_path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.677 DEBUG:boto:Method: GET 2024-04-09T21:52:44.677 DEBUG:boto:Path: /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.677 DEBUG:boto:Data: 2024-04-09T21:52:44.677 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.677 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.677 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.678 DEBUG:boto:Params: {} 2024-04-09T21:52:44.678 DEBUG:boto:Token: None 2024-04-09T21:52:44.678 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.678 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-09T21:52:44.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-09T21:52:44.680 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d972f6672f559ff4-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.681 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.681 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.681 DEBUG:boto:path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.681 DEBUG:boto:auth_path=/xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.681 DEBUG:boto:Method: GET 2024-04-09T21:52:44.681 DEBUG:boto:Path: /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.681 DEBUG:boto:Data: 2024-04-09T21:52:44.681 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.681 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.681 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.682 DEBUG:boto:Params: {} 2024-04-09T21:52:44.682 DEBUG:boto:Token: None 2024-04-09T21:52:44.682 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8?versionId=null 2024-04-09T21:52:44.682 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-09T21:52:44.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-09T21:52:44.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016663f98cc1a05da-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.684 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.684 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.684 DEBUG:boto:path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.684 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.684 DEBUG:boto:Method: GET 2024-04-09T21:52:44.684 DEBUG:boto:Path: /xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.684 DEBUG:boto:Data: 2024-04-09T21:52:44.684 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.685 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.685 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.685 DEBUG:boto:Params: {} 2024-04-09T21:52:44.685 DEBUG:boto:Token: None 2024-04-09T21:52:44.685 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.685 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3118500ba23c0c0-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.687 DEBUG:boto:b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.688 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.688 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.688 DEBUG:boto:path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.688 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.688 DEBUG:boto:Method: GET 2024-04-09T21:52:44.688 DEBUG:boto:Path: /xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.688 DEBUG:boto:Data: 2024-04-09T21:52:44.688 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.688 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.689 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.689 DEBUG:boto:Params: {} 2024-04-09T21:52:44.689 DEBUG:boto:Token: None 2024-04-09T21:52:44.689 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.689 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960b6a9a36dc459c-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.692 DEBUG:boto:b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.692 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.692 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.692 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.692 DEBUG:boto:Path: /xryqqk-11/obj8 2024-04-09T21:52:44.692 DEBUG:boto:Data: 2024-04-09T21:52:44.693 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.693 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.693 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.693 DEBUG:boto:Params: {} 2024-04-09T21:52:44.693 DEBUG:boto:Token: None 2024-04-09T21:52:44.693 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8 2024-04-09T21:52:44.693 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g= 2024-04-09T21:52:44.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g=', 'Content-Length': '0'} 2024-04-09T21:52:44.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000867fecc7d726ddab-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.695 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.695 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.695 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.695 DEBUG:boto:Path: /xryqqk-11/obj8 2024-04-09T21:52:44.695 DEBUG:boto:Data: 2024-04-09T21:52:44.695 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.695 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.695 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.695 DEBUG:boto:Params: {} 2024-04-09T21:52:44.695 DEBUG:boto:Token: None 2024-04-09T21:52:44.696 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8 2024-04-09T21:52:44.696 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g= 2024-04-09T21:52:44.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g=', 'Content-Length': '0'} 2024-04-09T21:52:44.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb366880f53fbd8-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-09T21:52:44.698 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.698 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.698 DEBUG:boto:Method: GET 2024-04-09T21:52:44.698 DEBUG:boto:Path: /xryqqk-11/obj8 2024-04-09T21:52:44.698 DEBUG:boto:Data: 2024-04-09T21:52:44.698 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.698 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.698 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.698 DEBUG:boto:Params: {} 2024-04-09T21:52:44.698 DEBUG:boto:Token: None 2024-04-09T21:52:44.699 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8 2024-04-09T21:52:44.699 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8= 2024-04-09T21:52:44.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8=', 'Content-Length': '0'} 2024-04-09T21:52:44.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1757000e445a681-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.746 DEBUG:boto:path=/xryqqk-11/obj8 2024-04-09T21:52:44.747 DEBUG:boto:auth_path=/xryqqk-11/obj8 2024-04-09T21:52:44.747 DEBUG:boto:Method: GET 2024-04-09T21:52:44.747 DEBUG:boto:Path: /xryqqk-11/obj8 2024-04-09T21:52:44.747 DEBUG:boto:Data: 2024-04-09T21:52:44.747 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.747 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.747 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.747 DEBUG:boto:Params: {} 2024-04-09T21:52:44.747 DEBUG:boto:Token: None 2024-04-09T21:52:44.748 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj8 2024-04-09T21:52:44.748 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8= 2024-04-09T21:52:44.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8=', 'Content-Length': '0'} 2024-04-09T21:52:44.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000492f99d3037ac1a8-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.750 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.750 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.750 DEBUG:boto:path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.750 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.750 DEBUG:boto:Method: GET 2024-04-09T21:52:44.750 DEBUG:boto:Path: /xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.751 DEBUG:boto:Data: 2024-04-09T21:52:44.751 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.751 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.751 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.751 DEBUG:boto:Params: {} 2024-04-09T21:52:44.751 DEBUG:boto:Token: None 2024-04-09T21:52:44.751 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.752 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb51a34cec2c6896-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.754 DEBUG:boto:b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.754 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.754 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.755 DEBUG:boto:path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.755 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.755 DEBUG:boto:Method: GET 2024-04-09T21:52:44.755 DEBUG:boto:Path: /xryqqk-11/?prefix=obj8 2024-04-09T21:52:44.755 DEBUG:boto:Data: 2024-04-09T21:52:44.755 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.755 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.755 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.755 DEBUG:boto:Params: {} 2024-04-09T21:52:44.755 DEBUG:boto:Token: None 2024-04-09T21:52:44.756 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.756 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2f6b1e5585eed3-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.758 DEBUG:boto:b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-04-09T21:52:44.759 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.759 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.759 DEBUG:boto:path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.759 DEBUG:boto:auth_path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.759 DEBUG:boto:Method: GET 2024-04-09T21:52:44.759 DEBUG:boto:Path: /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.759 DEBUG:boto:Data: 2024-04-09T21:52:44.759 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.759 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.759 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.759 DEBUG:boto:Params: {} 2024-04-09T21:52:44.760 DEBUG:boto:Token: None 2024-04-09T21:52:44.760 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.760 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-09T21:52:44.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-09T21:52:44.761 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34efa8a1f0bbe97-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.762 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.762 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.762 DEBUG:boto:path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.762 DEBUG:boto:auth_path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.762 DEBUG:boto:Method: GET 2024-04-09T21:52:44.762 DEBUG:boto:Path: /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.762 DEBUG:boto:Data: 2024-04-09T21:52:44.762 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.762 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.763 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.763 DEBUG:boto:Params: {} 2024-04-09T21:52:44.763 DEBUG:boto:Token: None 2024-04-09T21:52:44.763 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.763 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-09T21:52:44.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-09T21:52:44.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b56d1eb148427b18-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.765 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.765 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.765 DEBUG:boto:path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.765 DEBUG:boto:auth_path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.765 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.765 DEBUG:boto:Path: /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.765 DEBUG:boto:Data: 2024-04-09T21:52:44.766 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.766 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.766 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.766 DEBUG:boto:Params: {} 2024-04-09T21:52:44.766 DEBUG:boto:Token: None 2024-04-09T21:52:44.766 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.766 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM= 2024-04-09T21:52:44.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM=', 'Content-Length': '0'} 2024-04-09T21:52:44.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091f2eac27e5bc4f8-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.768 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.768 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.768 DEBUG:boto:path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.768 DEBUG:boto:auth_path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.768 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.769 DEBUG:boto:Path: /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.769 DEBUG:boto:Data: 2024-04-09T21:52:44.769 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.769 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.769 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.769 DEBUG:boto:Params: {} 2024-04-09T21:52:44.769 DEBUG:boto:Token: None 2024-04-09T21:52:44.769 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.769 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM= 2024-04-09T21:52:44.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM=', 'Content-Length': '0'} 2024-04-09T21:52:44.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061085883d1b68c48-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-04-09T21:52:44.771 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.772 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.772 DEBUG:boto:path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.772 DEBUG:boto:auth_path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.772 DEBUG:boto:Method: GET 2024-04-09T21:52:44.772 DEBUG:boto:Path: /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.772 DEBUG:boto:Data: 2024-04-09T21:52:44.772 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.772 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.772 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.772 DEBUG:boto:Params: {} 2024-04-09T21:52:44.772 DEBUG:boto:Token: None 2024-04-09T21:52:44.773 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.773 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-09T21:52:44.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-09T21:52:44.774 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e685693f676fa972-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.774 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.775 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.775 DEBUG:boto:path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.775 DEBUG:boto:auth_path=/xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.775 DEBUG:boto:Method: GET 2024-04-09T21:52:44.775 DEBUG:boto:Path: /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.775 DEBUG:boto:Data: 2024-04-09T21:52:44.775 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.775 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.775 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.775 DEBUG:boto:Params: {} 2024-04-09T21:52:44.775 DEBUG:boto:Token: None 2024-04-09T21:52:44.776 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9?versionId=null 2024-04-09T21:52:44.776 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-09T21:52:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-09T21:52:44.777 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000326fd855d00941e4-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.818 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.819 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.819 DEBUG:boto:path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.819 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.819 DEBUG:boto:Method: GET 2024-04-09T21:52:44.819 DEBUG:boto:Path: /xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.819 DEBUG:boto:Data: 2024-04-09T21:52:44.819 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.819 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.819 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.819 DEBUG:boto:Params: {} 2024-04-09T21:52:44.820 DEBUG:boto:Token: None 2024-04-09T21:52:44.820 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.820 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46303b64f883c79-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.822 DEBUG:boto:b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.823 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.823 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.823 DEBUG:boto:path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.823 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.823 DEBUG:boto:Method: GET 2024-04-09T21:52:44.823 DEBUG:boto:Path: /xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.823 DEBUG:boto:Data: 2024-04-09T21:52:44.823 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.824 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.824 DEBUG:boto:Params: {} 2024-04-09T21:52:44.824 DEBUG:boto:Token: None 2024-04-09T21:52:44.824 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063e2685bbadaaba5-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.826 DEBUG:boto:b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.826 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.827 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.827 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.827 DEBUG:boto:Path: /xryqqk-11/obj9 2024-04-09T21:52:44.827 DEBUG:boto:Data: 2024-04-09T21:52:44.827 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.827 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.827 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.827 DEBUG:boto:Params: {} 2024-04-09T21:52:44.827 DEBUG:boto:Token: None 2024-04-09T21:52:44.827 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9 2024-04-09T21:52:44.828 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc= 2024-04-09T21:52:44.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc=', 'Content-Length': '0'} 2024-04-09T21:52:44.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000828c5ecc11181c6b-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.829 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.829 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.829 DEBUG:boto:Method: HEAD 2024-04-09T21:52:44.829 DEBUG:boto:Path: /xryqqk-11/obj9 2024-04-09T21:52:44.830 DEBUG:boto:Data: 2024-04-09T21:52:44.830 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.830 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.830 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.830 DEBUG:boto:Params: {} 2024-04-09T21:52:44.830 DEBUG:boto:Token: None 2024-04-09T21:52:44.830 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9 2024-04-09T21:52:44.830 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc= 2024-04-09T21:52:44.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc=', 'Content-Length': '0'} 2024-04-09T21:52:44.832 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062ad1c9c1bafc68d-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj9 2024-04-09T21:52:44.832 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.832 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.832 DEBUG:boto:Method: GET 2024-04-09T21:52:44.833 DEBUG:boto:Path: /xryqqk-11/obj9 2024-04-09T21:52:44.833 DEBUG:boto:Data: 2024-04-09T21:52:44.833 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.833 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.833 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.833 DEBUG:boto:Params: {} 2024-04-09T21:52:44.833 DEBUG:boto:Token: None 2024-04-09T21:52:44.833 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9 2024-04-09T21:52:44.833 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI= 2024-04-09T21:52:44.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI=', 'Content-Length': '0'} 2024-04-09T21:52:44.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f71c034dc3cdae27-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.835 DEBUG:boto:path=/xryqqk-11/obj9 2024-04-09T21:52:44.835 DEBUG:boto:auth_path=/xryqqk-11/obj9 2024-04-09T21:52:44.835 DEBUG:boto:Method: GET 2024-04-09T21:52:44.835 DEBUG:boto:Path: /xryqqk-11/obj9 2024-04-09T21:52:44.836 DEBUG:boto:Data: 2024-04-09T21:52:44.836 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.836 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.836 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.836 DEBUG:boto:Params: {} 2024-04-09T21:52:44.836 DEBUG:boto:Token: None 2024-04-09T21:52:44.836 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/obj9 2024-04-09T21:52:44.836 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI= 2024-04-09T21:52:44.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI=', 'Content-Length': '0'} 2024-04-09T21:52:44.838 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d1af103f3207fb9-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.838 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.838 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.838 DEBUG:boto:path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.838 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.839 DEBUG:boto:Method: GET 2024-04-09T21:52:44.839 DEBUG:boto:Path: /xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.839 DEBUG:boto:Data: 2024-04-09T21:52:44.839 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.839 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:52:44.839 DEBUG:boto:Port: 8001 2024-04-09T21:52:44.839 DEBUG:boto:Params: {} 2024-04-09T21:52:44.839 DEBUG:boto:Token: None 2024-04-09T21:52:44.839 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.839 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113249aa343b9c6d-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.882 DEBUG:boto:b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.883 DEBUG:boto:path=/xryqqk-11/ 2024-04-09T21:52:44.883 DEBUG:boto:auth_path=/xryqqk-11/ 2024-04-09T21:52:44.883 DEBUG:boto:path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.883 DEBUG:boto:auth_path=/xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.883 DEBUG:boto:Method: GET 2024-04-09T21:52:44.883 DEBUG:boto:Path: /xryqqk-11/?prefix=obj9 2024-04-09T21:52:44.883 DEBUG:boto:Data: 2024-04-09T21:52:44.884 DEBUG:boto:Headers: {} 2024-04-09T21:52:44.884 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:52:44.884 DEBUG:boto:Port: 8000 2024-04-09T21:52:44.884 DEBUG:boto:Params: {} 2024-04-09T21:52:44.884 DEBUG:boto:Token: None 2024-04-09T21:52:44.884 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:52:44 GMT /xryqqk-11/ 2024-04-09T21:52:44.884 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-09T21:52:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-09T21:52:44.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13ec758fc2b816f-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:52:44.887 DEBUG:boto:b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:52:44.887 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-11 zones={a2, a1} 2024-04-09T21:52:44.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:52:44.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:52:44.888 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:52:44.946 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:52:44.946 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:52:44.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:44.981+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:52:49.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:49.981+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:52:54.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:54.982+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:52:59.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:52:59.983+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:04.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:04.983+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:09.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:09.984+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:14.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:14.985+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:19.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:19.986+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:24.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:24.987+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:29.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:29.986+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:34.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:34.987+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:39.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:39.987+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:44.988+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:49.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:49.989+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:54.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:54.989+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:53:59.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:53:59.990+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:04.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:04.991+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:09.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:09.991+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:14.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:14.992+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:19.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:19.993+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:24.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:24.993+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:29.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:29.994+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:34.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:34.994+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:39.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:39.995+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:44.995+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:49.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:49.997+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:54.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:54.997+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:54:59.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:54:59.998+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:04.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:04.999+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:10.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:09.998+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:15.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:14.999+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:20.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:20.000+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:25.000+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:30.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:30.001+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:35.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:35.001+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:40.002+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:45.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:45.003+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:50.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:50.003+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:55:55.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:55:55.004+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:00.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:00.005+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:05.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:05.005+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:10.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:10.006+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:15.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:15.006+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:20.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:20.007+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:25.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:25.008+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:30.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:30.008+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:35.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:35.010+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:40.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:40.010+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:45.010+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:50.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:50.011+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:56:55.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:56:55.011+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:00.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:00.012+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:05.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:05.012+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:10.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:10.013+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:15.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:15.014+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:20.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:20.014+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:25.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:25.015+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:30.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:30.016+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:35.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:35.016+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-09T21:57:40.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:40.017+0000 7f8bdefdb780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T21:57:40.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:40.017+0000 7f8bdefdb780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T21:57:40.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:40.017+0000 7f8bdefdb780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T21:57:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T21:57:40.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-04-09T21:57:40.066 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-13 2024-04-09T21:57:40.067 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:57:40.067 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:57:40.067 DEBUG:boto:Method: PUT 2024-04-09T21:57:40.067 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:57:40.067 DEBUG:boto:Data: 2024-04-09T21:57:40.067 DEBUG:boto:Headers: {} 2024-04-09T21:57:40.067 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:57:40.067 DEBUG:boto:Port: 8000 2024-04-09T21:57:40.067 DEBUG:boto:Params: {} 2024-04-09T21:57:40.068 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T21:57:40.068 DEBUG:boto:Token: None 2024-04-09T21:57:40.068 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:57:40 GMT /xryqqk-13/ 2024-04-09T21:57:40.068 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DiQgq3ZKjzVvcaocvv6IQidI1Dk= 2024-04-09T21:57:40.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:57:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DiQgq3ZKjzVvcaocvv6IQidI1Dk=', 'Content-Length': '0'} 2024-04-09T21:57:40.087 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d8624b5ad912523-006615b9d4-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:57:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:57:40.087 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-14 2024-04-09T21:57:40.087 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:57:40.087 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:57:40.087 DEBUG:boto:Method: PUT 2024-04-09T21:57:40.088 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:57:40.088 DEBUG:boto:Data: 2024-04-09T21:57:40.088 DEBUG:boto:Headers: {} 2024-04-09T21:57:40.088 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:57:40.088 DEBUG:boto:Port: 8001 2024-04-09T21:57:40.088 DEBUG:boto:Params: {} 2024-04-09T21:57:40.088 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T21:57:40.088 DEBUG:boto:Token: None 2024-04-09T21:57:40.088 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:57:40 GMT /xryqqk-14/ 2024-04-09T21:57:40.088 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rxt3kFXywtawTvi1eV4rjOwhNow= 2024-04-09T21:57:40.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:57:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rxt3kFXywtawTvi1eV4rjOwhNow=', 'Content-Length': '0'} 2024-04-09T21:57:40.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008eb419e6e9530fb1-006615b9d4-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:57:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:57:40.119 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:57:40.119 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:57:40.119 DEBUG:boto:path=/xryqqk-13/?versioning 2024-04-09T21:57:40.119 DEBUG:boto:auth_path=/xryqqk-13/?versioning 2024-04-09T21:57:40.119 DEBUG:boto:Method: PUT 2024-04-09T21:57:40.120 DEBUG:boto:Path: /xryqqk-13/?versioning 2024-04-09T21:57:40.120 DEBUG:boto:Data: Enabled Disabled 2024-04-09T21:57:40.120 DEBUG:boto:Headers: {} 2024-04-09T21:57:40.120 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:57:40.120 DEBUG:boto:Port: 8000 2024-04-09T21:57:40.120 DEBUG:boto:Params: {} 2024-04-09T21:57:40.120 DEBUG:boto:Token: None 2024-04-09T21:57:40.120 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:57:40 GMT /xryqqk-13/?versioning 2024-04-09T21:57:40.120 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:x9CAWxhllNpLeKn2uCD7HgEuIkE= 2024-04-09T21:57:40.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:57:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x9CAWxhllNpLeKn2uCD7HgEuIkE=', 'Content-Length': '228'} 2024-04-09T21:57:40.126 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000010e5962cde36c2fd-006615b9d4-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:57:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:57:40.126 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:57:40.126 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:57:40.126 DEBUG:boto:path=/xryqqk-14/?versioning 2024-04-09T21:57:40.126 DEBUG:boto:auth_path=/xryqqk-14/?versioning 2024-04-09T21:57:40.126 DEBUG:boto:Method: PUT 2024-04-09T21:57:40.126 DEBUG:boto:Path: /xryqqk-14/?versioning 2024-04-09T21:57:40.126 DEBUG:boto:Data: Enabled Disabled 2024-04-09T21:57:40.127 DEBUG:boto:Headers: {} 2024-04-09T21:57:40.127 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:57:40.127 DEBUG:boto:Port: 8001 2024-04-09T21:57:40.127 DEBUG:boto:Params: {} 2024-04-09T21:57:40.127 DEBUG:boto:Token: None 2024-04-09T21:57:40.127 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:57:40 GMT /xryqqk-14/?versioning 2024-04-09T21:57:40.127 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YRvxJBds9xbElplt7QHfXd+NSNc= 2024-04-09T21:57:40.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:57:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YRvxJBds9xbElplt7QHfXd+NSNc=', 'Content-Length': '228'} 2024-04-09T21:57:40.135 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002de84ecf257b6738-006615b9d4-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:57:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:57:40.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:57:40.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:57:40.136 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:57:40.187 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:57:40.187 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:57:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:57:40.230 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699860.132444_202.1", 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:57:40.132444Z" 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:40.232 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:57:40.233 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699860.132444_202.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:57:40.233 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:57:40.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:57:40.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:57:40.234 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:57:40.289 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:57:40.289 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:57:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:40.324+0000 7f46362f3780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:57:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:40.325+0000 7f46362f3780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:57:40.337 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:57:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:57:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:57:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:57:40.337 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:57:40.338 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:57:40.339 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699860.129613_201.1", 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:40.340 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:57:40.129613Z", 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:57:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:57:40.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:40.343 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:57:40Z" 2024-04-09T21:57:40.343 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:57:40.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699860.129613_201.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:57:40.129613Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:57:40Z" } 2024-04-09T21:57:40.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699860.129613_201.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:57:40.129613Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:57:40.344 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699860.132444_202.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:57:40.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699860.129613_201.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:57:40.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712699860.132444_202.1 target=1_1712699860.129613_201.1 2024-04-09T21:57:45.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:57:45.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:57:45.345 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:57:45.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:57:45.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:57:45.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:45.436+0000 7f9ff27ae780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:57:45.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:45.437+0000 7f9ff27ae780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:57:45.451 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:57:45.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:45.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:45.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699860.132444_202.1", 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:57:40.132444Z", 2024-04-09T21:57:45.455 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:57:45.456 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:57:45Z" 2024-04-09T21:57:45.457 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:57:45.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699860.132444_202.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:57:40.132444Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:57:45Z" } 2024-04-09T21:57:45.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699860.132444_202.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:57:40.132444Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:57:45.459 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699860.132444_202.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:57:45.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699860.132444_202.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:57:45.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:57:45.459 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:57:45.459 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:57:45.459 DEBUG:boto:Method: HEAD 2024-04-09T21:57:45.459 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:57:45.459 DEBUG:boto:Data: 2024-04-09T21:57:45.460 DEBUG:boto:Headers: {} 2024-04-09T21:57:45.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:57:45.460 DEBUG:boto:Port: 8000 2024-04-09T21:57:45.460 DEBUG:boto:Params: {} 2024-04-09T21:57:45.460 DEBUG:boto:Token: None 2024-04-09T21:57:45.460 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:57:45 GMT /xryqqk-13/ 2024-04-09T21:57:45.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+ewtg6GmenMuViTcPcvPWZ4Ak2U= 2024-04-09T21:57:45.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:57:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+ewtg6GmenMuViTcPcvPWZ4Ak2U=', 'Content-Length': '0'} 2024-04-09T21:57:45.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098c6345a9617d9e8-006615b9d9-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:57:45.464 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:57:45.464 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:57:45.464 DEBUG:boto:Method: PUT 2024-04-09T21:57:45.464 DEBUG:boto:Path: /xryqqk-13/dummy 2024-04-09T21:57:45.465 DEBUG:boto:Data: 2024-04-09T21:57:45.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T21:57:45.465 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:57:45.465 DEBUG:boto:Port: 8000 2024-04-09T21:57:45.465 DEBUG:boto:Params: {} 2024-04-09T21:57:45.465 DEBUG:boto:Token: None 2024-04-09T21:57:45.465 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 21:57:45 GMT /xryqqk-13/dummy 2024-04-09T21:57:45.465 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+PAcCuBE/oG6kjRoAx1NR5Bwh+k= 2024-04-09T21:57:45.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:57:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+PAcCuBE/oG6kjRoAx1NR5Bwh+k='} 2024-04-09T21:57:45.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-amz-request-id', 'tx0000082ee589e5040c983-006615b9d9-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:57:45.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:57:45.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:57:45.477 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:57:45.535 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:57:45.535 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:57:45.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:45.569+0000 7fe83fb29780 1 waiting to reach incremental sync.. 2024-04-09T21:57:50.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:50.570+0000 7fe83fb29780 1 waiting to reach incremental sync.. 2024-04-09T21:57:55.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:57:55.571+0000 7fe83fb29780 1 waiting to reach incremental sync.. 2024-04-09T21:58:00.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:00.573+0000 7fe83fb29780 1 bucket sync caught up with source: 2024-04-09T21:58:00.574 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2024-04-09T21:58:00.574 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.5.11, , , , , ] 2024-04-09T21:58:00.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:00.573+0000 7fe83fb29780 0 bucket checkpoint complete 2024-04-09T21:58:00.591 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:58:00.591 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:58:00.591 DEBUG:boto:Method: HEAD 2024-04-09T21:58:00.591 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:58:00.591 DEBUG:boto:Data: 2024-04-09T21:58:00.591 DEBUG:boto:Headers: {} 2024-04-09T21:58:00.592 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:00.592 DEBUG:boto:Port: 8001 2024-04-09T21:58:00.592 DEBUG:boto:Params: {} 2024-04-09T21:58:00.592 DEBUG:boto:Token: None 2024-04-09T21:58:00.592 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:58:00 GMT /xryqqk-14/ 2024-04-09T21:58:00.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4uYjnbDww1c3pjLyJFCEClZlpaI= 2024-04-09T21:58:00.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4uYjnbDww1c3pjLyJFCEClZlpaI=', 'Content-Length': '0'} 2024-04-09T21:58:00.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010476e9fc2422c99-006615b9e8-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:00.595 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:58:00.595 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:58:00.595 DEBUG:boto:Method: PUT 2024-04-09T21:58:00.596 DEBUG:boto:Path: /xryqqk-14/dummy 2024-04-09T21:58:00.596 DEBUG:boto:Data: 2024-04-09T21:58:00.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T21:58:00.596 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:00.596 DEBUG:boto:Port: 8001 2024-04-09T21:58:00.596 DEBUG:boto:Params: {} 2024-04-09T21:58:00.596 DEBUG:boto:Token: None 2024-04-09T21:58:00.596 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 21:58:00 GMT /xryqqk-14/dummy 2024-04-09T21:58:00.596 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LuRsvFDxlH/tbcYH+kb4oGsf5+E= 2024-04-09T21:58:00.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LuRsvFDxlH/tbcYH+kb4oGsf5+E='} 2024-04-09T21:58:00.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-amz-request-id', 'tx000004574c4e7d484af41-006615b9e8-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:00.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:58:00.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:58:00.609 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:58:00.670 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:58:00.671 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:58:00.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:00.706+0000 7fcf1cd90780 1 waiting to reach incremental sync.. 2024-04-09T21:58:05.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:05.707+0000 7fcf1cd90780 1 waiting to reach incremental sync.. 2024-04-09T21:58:10.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:10.708+0000 7fcf1cd90780 1 waiting to reach incremental sync.. 2024-04-09T21:58:15.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:15.709+0000 7fcf1cd90780 1 waiting to reach incremental sync.. 2024-04-09T21:58:20.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:20.710+0000 7fcf1cd90780 1 bucket sync caught up with source: 2024-04-09T21:58:20.711 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.16.11, , , , , ] 2024-04-09T21:58:20.711 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.16.11, , , , , ] 2024-04-09T21:58:20.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:20.710+0000 7fcf1cd90780 0 bucket checkpoint complete 2024-04-09T21:58:20.728 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:58:20.728 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:58:20.728 DEBUG:boto:Method: HEAD 2024-04-09T21:58:20.728 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:58:20.728 DEBUG:boto:Data: 2024-04-09T21:58:20.728 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.728 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.728 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.728 DEBUG:boto:Params: {} 2024-04-09T21:58:20.729 DEBUG:boto:Token: None 2024-04-09T21:58:20.729 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/ 2024-04-09T21:58:20.729 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:USrrkrKj9l62jZkv8M7MQcPfJm0= 2024-04-09T21:58:20.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:USrrkrKj9l62jZkv8M7MQcPfJm0=', 'Content-Length': '0'} 2024-04-09T21:58:20.732 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d58f53be136f3df-006615b9fc-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.732 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.732 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.732 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.732 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:58:20.732 DEBUG:boto:Data: 2024-04-09T21:58:20.733 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.733 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.733 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.733 DEBUG:boto:Params: {} 2024-04-09T21:58:20.733 DEBUG:boto:Token: None 2024-04-09T21:58:20.733 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a1 2024-04-09T21:58:20.733 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:BcPT2mbBTcclwCzSMOMPTo2srQM= 2024-04-09T21:58:20.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BcPT2mbBTcclwCzSMOMPTo2srQM='} 2024-04-09T21:58:20.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-amz-request-id', 'tx000006711ed10777faa3c-006615b9fc-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.775 DEBUG:rgw_multi.tests:version1 id=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:58:20.775 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.775 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.775 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.775 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:58:20.775 DEBUG:boto:Data: 2024-04-09T21:58:20.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.776 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.776 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.776 DEBUG:boto:Params: {} 2024-04-09T21:58:20.776 DEBUG:boto:Token: None 2024-04-09T21:58:20.776 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a1 2024-04-09T21:58:20.776 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:A0fs8S3uLeE5JfLk3Cw9SbzHh7I= 2024-04-09T21:58:20.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A0fs8S3uLeE5JfLk3Cw9SbzHh7I='} 2024-04-09T21:58:20.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6j6.OdmbPjavO3BwVfdLwburuVMXGjj'), ('x-amz-request-id', 'tx0000039d550fd601be065-006615b9fc-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.788 DEBUG:rgw_multi.tests:version2 id=6j6.OdmbPjavO3BwVfdLwburuVMXGjj 2024-04-09T21:58:20.788 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.788 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.789 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=6j6.OdmbPjavO3BwVfdLwburuVMXGjj 2024-04-09T21:58:20.789 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=6j6.OdmbPjavO3BwVfdLwburuVMXGjj 2024-04-09T21:58:20.789 DEBUG:boto:Method: DELETE 2024-04-09T21:58:20.789 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=6j6.OdmbPjavO3BwVfdLwburuVMXGjj 2024-04-09T21:58:20.789 DEBUG:boto:Data: 2024-04-09T21:58:20.789 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.789 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.789 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.789 DEBUG:boto:Params: {} 2024-04-09T21:58:20.789 DEBUG:boto:Token: None 2024-04-09T21:58:20.789 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a1?versionId=6j6.OdmbPjavO3BwVfdLwburuVMXGjj 2024-04-09T21:58:20.790 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:D4pV5ZMWerwbGE5OpJY0kco4df8= 2024-04-09T21:58:20.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:D4pV5ZMWerwbGE5OpJY0kco4df8=', 'Content-Length': '0'} 2024-04-09T21:58:20.799 DEBUG:boto:Response headers: [('x-amz-version-id', '6j6.OdmbPjavO3BwVfdLwburuVMXGjj'), ('x-amz-request-id', 'tx00000b12f34d05faeb8b4-006615b9fc-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.800 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.800 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.800 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.800 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:58:20.800 DEBUG:boto:Data: 2024-04-09T21:58:20.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.800 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.800 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.800 DEBUG:boto:Params: {} 2024-04-09T21:58:20.800 DEBUG:boto:Token: None 2024-04-09T21:58:20.801 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a1 2024-04-09T21:58:20.801 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gA6dNH5CsxxN3cMriXCeFSIiY/c= 2024-04-09T21:58:20.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gA6dNH5CsxxN3cMriXCeFSIiY/c='} 2024-04-09T21:58:20.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6'), ('x-amz-request-id', 'tx00000fa46add983686039-006615b9fc-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.843 DEBUG:rgw_multi.tests:version3 id=p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6 2024-04-09T21:58:20.843 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.843 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:58:20.843 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6 2024-04-09T21:58:20.843 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6 2024-04-09T21:58:20.843 DEBUG:boto:Method: DELETE 2024-04-09T21:58:20.843 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6 2024-04-09T21:58:20.843 DEBUG:boto:Data: 2024-04-09T21:58:20.843 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.843 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.843 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.843 DEBUG:boto:Params: {} 2024-04-09T21:58:20.844 DEBUG:boto:Token: None 2024-04-09T21:58:20.844 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a1?versionId=p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6 2024-04-09T21:58:20.844 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pdnuoZdkW7GacL3IXWGWKqfjwX0= 2024-04-09T21:58:20.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pdnuoZdkW7GacL3IXWGWKqfjwX0=', 'Content-Length': '0'} 2024-04-09T21:58:20.855 DEBUG:boto:Response headers: [('x-amz-version-id', 'p2M1WgLR72UPQ.GDZcxIuA-EgYruvN6'), ('x-amz-request-id', 'tx000002357ed2ab5c7f555-006615b9fc-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.855 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:58:20.855 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:58:20.855 DEBUG:boto:Method: HEAD 2024-04-09T21:58:20.855 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:58:20.856 DEBUG:boto:Data: 2024-04-09T21:58:20.856 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.856 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:20.856 DEBUG:boto:Port: 8001 2024-04-09T21:58:20.856 DEBUG:boto:Params: {} 2024-04-09T21:58:20.856 DEBUG:boto:Token: None 2024-04-09T21:58:20.856 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/ 2024-04-09T21:58:20.856 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:USrrkrKj9l62jZkv8M7MQcPfJm0= 2024-04-09T21:58:20.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:USrrkrKj9l62jZkv8M7MQcPfJm0=', 'Content-Length': '0'} 2024-04-09T21:58:20.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000eee3d3d3a343b98-006615b9fc-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.859 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.859 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.859 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.859 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:58:20.859 DEBUG:boto:Data: 2024-04-09T21:58:20.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.859 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:20.859 DEBUG:boto:Port: 8001 2024-04-09T21:58:20.859 DEBUG:boto:Params: {} 2024-04-09T21:58:20.860 DEBUG:boto:Token: None 2024-04-09T21:58:20.860 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a2 2024-04-09T21:58:20.860 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9woNSKszmv0hdPuVoPKyVpA1jtI= 2024-04-09T21:58:20.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9woNSKszmv0hdPuVoPKyVpA1jtI='} 2024-04-09T21:58:20.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-amz-request-id', 'tx00000d0dd5875a4431931-006615b9fc-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.903 DEBUG:rgw_multi.tests:version1 id=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:58:20.903 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.903 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.903 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.903 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:58:20.903 DEBUG:boto:Data: 2024-04-09T21:58:20.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.904 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:20.904 DEBUG:boto:Port: 8001 2024-04-09T21:58:20.904 DEBUG:boto:Params: {} 2024-04-09T21:58:20.904 DEBUG:boto:Token: None 2024-04-09T21:58:20.904 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a2 2024-04-09T21:58:20.904 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IPXgHMrbpbjD9WicPQHeNC7et64= 2024-04-09T21:58:20.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IPXgHMrbpbjD9WicPQHeNC7et64='} 2024-04-09T21:58:20.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FQt8xt93yPoBye5NL58eOKlVlR9ayQu'), ('x-amz-request-id', 'tx0000099726206e33c53f6-006615b9fc-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.913 DEBUG:rgw_multi.tests:version2 id=FQt8xt93yPoBye5NL58eOKlVlR9ayQu 2024-04-09T21:58:20.913 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.913 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.914 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=FQt8xt93yPoBye5NL58eOKlVlR9ayQu 2024-04-09T21:58:20.914 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=FQt8xt93yPoBye5NL58eOKlVlR9ayQu 2024-04-09T21:58:20.914 DEBUG:boto:Method: DELETE 2024-04-09T21:58:20.914 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=FQt8xt93yPoBye5NL58eOKlVlR9ayQu 2024-04-09T21:58:20.914 DEBUG:boto:Data: 2024-04-09T21:58:20.914 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.914 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:20.914 DEBUG:boto:Port: 8001 2024-04-09T21:58:20.914 DEBUG:boto:Params: {} 2024-04-09T21:58:20.914 DEBUG:boto:Token: None 2024-04-09T21:58:20.914 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a2?versionId=FQt8xt93yPoBye5NL58eOKlVlR9ayQu 2024-04-09T21:58:20.915 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fLG1MjRw1QhPife/q8iWrbpRiT8= 2024-04-09T21:58:20.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fLG1MjRw1QhPife/q8iWrbpRiT8=', 'Content-Length': '0'} 2024-04-09T21:58:20.924 DEBUG:boto:Response headers: [('x-amz-version-id', 'FQt8xt93yPoBye5NL58eOKlVlR9ayQu'), ('x-amz-request-id', 'tx00000bf3a7930b2edec81-006615b9fc-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.924 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.924 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.924 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.925 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:58:20.925 DEBUG:boto:Data: 2024-04-09T21:58:20.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.925 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:20.925 DEBUG:boto:Port: 8001 2024-04-09T21:58:20.925 DEBUG:boto:Params: {} 2024-04-09T21:58:20.925 DEBUG:boto:Token: None 2024-04-09T21:58:20.925 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a2 2024-04-09T21:58:20.925 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:BCY3AiJqT3EZfbAk2IdaySZ6iHY= 2024-04-09T21:58:20.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BCY3AiJqT3EZfbAk2IdaySZ6iHY='} 2024-04-09T21:58:20.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5'), ('x-amz-request-id', 'tx00000c1d8e554060f9d9b-006615b9fc-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.975 DEBUG:rgw_multi.tests:version3 id=MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5 2024-04-09T21:58:20.975 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.975 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:58:20.975 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5 2024-04-09T21:58:20.975 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5 2024-04-09T21:58:20.976 DEBUG:boto:Method: DELETE 2024-04-09T21:58:20.976 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5 2024-04-09T21:58:20.976 DEBUG:boto:Data: 2024-04-09T21:58:20.976 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.976 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:20.976 DEBUG:boto:Port: 8001 2024-04-09T21:58:20.976 DEBUG:boto:Params: {} 2024-04-09T21:58:20.977 DEBUG:boto:Token: None 2024-04-09T21:58:20.977 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-13/obj-a2?versionId=MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5 2024-04-09T21:58:20.977 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:eofAY84GWobpKrP7LX7qDZKTiVw= 2024-04-09T21:58:20.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eofAY84GWobpKrP7LX7qDZKTiVw=', 'Content-Length': '0'} 2024-04-09T21:58:20.986 DEBUG:boto:Response headers: [('x-amz-version-id', 'MPF-HJ.Ka3zsBdYmpuf5QmimpVhh6i5'), ('x-amz-request-id', 'tx0000065d11000e9a0d3f8-006615b9fc-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.986 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:58:20.987 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:58:20.987 DEBUG:boto:Method: HEAD 2024-04-09T21:58:20.987 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:58:20.987 DEBUG:boto:Data: 2024-04-09T21:58:20.987 DEBUG:boto:Headers: {} 2024-04-09T21:58:20.987 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.987 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.987 DEBUG:boto:Params: {} 2024-04-09T21:58:20.987 DEBUG:boto:Token: None 2024-04-09T21:58:20.988 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-14/ 2024-04-09T21:58:20.988 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1prLydI5d4LmfUdceaTkSF8fBT4= 2024-04-09T21:58:20.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1prLydI5d4LmfUdceaTkSF8fBT4=', 'Content-Length': '0'} 2024-04-09T21:58:20.990 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036a0c45a0ef38b58-006615b9fc-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:20.990 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:58:20.990 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:58:20.990 DEBUG:boto:Method: PUT 2024-04-09T21:58:20.991 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:58:20.991 DEBUG:boto:Data: 2024-04-09T21:58:20.991 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:20.991 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:20.991 DEBUG:boto:Port: 8000 2024-04-09T21:58:20.991 DEBUG:boto:Params: {} 2024-04-09T21:58:20.991 DEBUG:boto:Token: None 2024-04-09T21:58:20.991 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 21:58:20 GMT /xryqqk-14/obj-a1 2024-04-09T21:58:20.991 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ALb1Vq9zngPPH9L/EYhgiivmlG8= 2024-04-09T21:58:20.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:20 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ALb1Vq9zngPPH9L/EYhgiivmlG8='} 2024-04-09T21:58:21.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-amz-request-id', 'tx00000f7a19b3b472a21d7-006615b9fc-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.003 DEBUG:rgw_multi.tests:version1 id=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:58:21.003 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.003 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.004 DEBUG:boto:Method: PUT 2024-04-09T21:58:21.004 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:58:21.004 DEBUG:boto:Data: 2024-04-09T21:58:21.004 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:21.004 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:21.004 DEBUG:boto:Port: 8000 2024-04-09T21:58:21.004 DEBUG:boto:Params: {} 2024-04-09T21:58:21.004 DEBUG:boto:Token: None 2024-04-09T21:58:21.004 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a1 2024-04-09T21:58:21.005 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ycxSPbOhME6iME16rxgQM8mbdTA= 2024-04-09T21:58:21.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ycxSPbOhME6iME16rxgQM8mbdTA='} 2024-04-09T21:58:21.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp'), ('x-amz-request-id', 'tx00000a904610c98f02dab-006615b9fd-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.016 DEBUG:rgw_multi.tests:version2 id=zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp 2024-04-09T21:58:21.016 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.016 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.016 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp 2024-04-09T21:58:21.016 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp 2024-04-09T21:58:21.016 DEBUG:boto:Method: DELETE 2024-04-09T21:58:21.016 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp 2024-04-09T21:58:21.016 DEBUG:boto:Data: 2024-04-09T21:58:21.016 DEBUG:boto:Headers: {} 2024-04-09T21:58:21.016 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:21.017 DEBUG:boto:Port: 8000 2024-04-09T21:58:21.017 DEBUG:boto:Params: {} 2024-04-09T21:58:21.017 DEBUG:boto:Token: None 2024-04-09T21:58:21.017 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a1?versionId=zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp 2024-04-09T21:58:21.017 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HgUBi8jQJOeOU/z9tCxWTaYg+gM= 2024-04-09T21:58:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HgUBi8jQJOeOU/z9tCxWTaYg+gM=', 'Content-Length': '0'} 2024-04-09T21:58:21.027 DEBUG:boto:Response headers: [('x-amz-version-id', 'zjPbuQ.7TlXvGSUR0DlllX5jM5TXgRp'), ('x-amz-request-id', 'tx00000cb9cbe03108dd735-006615b9fd-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.027 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.028 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.028 DEBUG:boto:Method: PUT 2024-04-09T21:58:21.028 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:58:21.028 DEBUG:boto:Data: 2024-04-09T21:58:21.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:21.028 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:21.028 DEBUG:boto:Port: 8000 2024-04-09T21:58:21.028 DEBUG:boto:Params: {} 2024-04-09T21:58:21.028 DEBUG:boto:Token: None 2024-04-09T21:58:21.028 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a1 2024-04-09T21:58:21.028 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:a7tiYleeZuupvJsBr7IsXvWwjOE= 2024-04-09T21:58:21.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:a7tiYleeZuupvJsBr7IsXvWwjOE='} 2024-04-09T21:58:21.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UTOzul4pHbB664Jj.zXIuHu8dyOP8ks'), ('x-amz-request-id', 'tx0000084caa28a93c22f49-006615b9fd-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.071 DEBUG:rgw_multi.tests:version3 id=UTOzul4pHbB664Jj.zXIuHu8dyOP8ks 2024-04-09T21:58:21.071 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.071 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:58:21.071 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=UTOzul4pHbB664Jj.zXIuHu8dyOP8ks 2024-04-09T21:58:21.071 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=UTOzul4pHbB664Jj.zXIuHu8dyOP8ks 2024-04-09T21:58:21.071 DEBUG:boto:Method: DELETE 2024-04-09T21:58:21.071 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=UTOzul4pHbB664Jj.zXIuHu8dyOP8ks 2024-04-09T21:58:21.072 DEBUG:boto:Data: 2024-04-09T21:58:21.072 DEBUG:boto:Headers: {} 2024-04-09T21:58:21.072 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:58:21.072 DEBUG:boto:Port: 8000 2024-04-09T21:58:21.072 DEBUG:boto:Params: {} 2024-04-09T21:58:21.072 DEBUG:boto:Token: None 2024-04-09T21:58:21.072 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a1?versionId=UTOzul4pHbB664Jj.zXIuHu8dyOP8ks 2024-04-09T21:58:21.072 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rXNSi/IsnXOejltA9Z52fAv2yM0= 2024-04-09T21:58:21.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rXNSi/IsnXOejltA9Z52fAv2yM0=', 'Content-Length': '0'} 2024-04-09T21:58:21.081 DEBUG:boto:Response headers: [('x-amz-version-id', 'UTOzul4pHbB664Jj.zXIuHu8dyOP8ks'), ('x-amz-request-id', 'tx00000b7585bb9367d3960-006615b9fd-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.081 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:58:21.082 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:58:21.082 DEBUG:boto:Method: HEAD 2024-04-09T21:58:21.082 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:58:21.082 DEBUG:boto:Data: 2024-04-09T21:58:21.082 DEBUG:boto:Headers: {} 2024-04-09T21:58:21.082 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:21.082 DEBUG:boto:Port: 8001 2024-04-09T21:58:21.082 DEBUG:boto:Params: {} 2024-04-09T21:58:21.082 DEBUG:boto:Token: None 2024-04-09T21:58:21.082 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/ 2024-04-09T21:58:21.083 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Qxn/2fPX8l1XjHGixB+APl1MOAM= 2024-04-09T21:58:21.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Qxn/2fPX8l1XjHGixB+APl1MOAM=', 'Content-Length': '0'} 2024-04-09T21:58:21.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a7b097c3d6ce74e-006615b9fd-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.085 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.085 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.085 DEBUG:boto:Method: PUT 2024-04-09T21:58:21.085 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:58:21.085 DEBUG:boto:Data: 2024-04-09T21:58:21.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:21.086 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:21.086 DEBUG:boto:Port: 8001 2024-04-09T21:58:21.086 DEBUG:boto:Params: {} 2024-04-09T21:58:21.086 DEBUG:boto:Token: None 2024-04-09T21:58:21.086 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a2 2024-04-09T21:58:21.086 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:td6CDLntrWwYxOe6znfP4H2eZJo= 2024-04-09T21:58:21.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:td6CDLntrWwYxOe6znfP4H2eZJo='} 2024-04-09T21:58:21.099 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-amz-request-id', 'tx00000027d387dcc1ad0fa-006615b9fd-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.099 DEBUG:rgw_multi.tests:version1 id=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:58:21.100 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.100 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.100 DEBUG:boto:Method: PUT 2024-04-09T21:58:21.100 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:58:21.100 DEBUG:boto:Data: 2024-04-09T21:58:21.100 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:21.100 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:21.100 DEBUG:boto:Port: 8001 2024-04-09T21:58:21.100 DEBUG:boto:Params: {} 2024-04-09T21:58:21.100 DEBUG:boto:Token: None 2024-04-09T21:58:21.101 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a2 2024-04-09T21:58:21.101 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:D1kE/t8QcSY7NHficNq1YAkO6cw= 2024-04-09T21:58:21.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:D1kE/t8QcSY7NHficNq1YAkO6cw='} 2024-04-09T21:58:21.113 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'jJ8xXJBFR4WSX5txZNbEarjDdPtOETj'), ('x-amz-request-id', 'tx00000a67c5dbcc3ed45f2-006615b9fd-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.113 DEBUG:rgw_multi.tests:version2 id=jJ8xXJBFR4WSX5txZNbEarjDdPtOETj 2024-04-09T21:58:21.113 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.113 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.113 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=jJ8xXJBFR4WSX5txZNbEarjDdPtOETj 2024-04-09T21:58:21.113 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=jJ8xXJBFR4WSX5txZNbEarjDdPtOETj 2024-04-09T21:58:21.113 DEBUG:boto:Method: DELETE 2024-04-09T21:58:21.113 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=jJ8xXJBFR4WSX5txZNbEarjDdPtOETj 2024-04-09T21:58:21.113 DEBUG:boto:Data: 2024-04-09T21:58:21.113 DEBUG:boto:Headers: {} 2024-04-09T21:58:21.114 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:21.114 DEBUG:boto:Port: 8001 2024-04-09T21:58:21.114 DEBUG:boto:Params: {} 2024-04-09T21:58:21.114 DEBUG:boto:Token: None 2024-04-09T21:58:21.114 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a2?versionId=jJ8xXJBFR4WSX5txZNbEarjDdPtOETj 2024-04-09T21:58:21.114 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mg6y/OsDj5cThiXme6KEQTpigSI= 2024-04-09T21:58:21.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mg6y/OsDj5cThiXme6KEQTpigSI=', 'Content-Length': '0'} 2024-04-09T21:58:21.123 DEBUG:boto:Response headers: [('x-amz-version-id', 'jJ8xXJBFR4WSX5txZNbEarjDdPtOETj'), ('x-amz-request-id', 'tx000003f4edd5728108607-006615b9fd-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.123 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.123 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.123 DEBUG:boto:Method: PUT 2024-04-09T21:58:21.124 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:58:21.124 DEBUG:boto:Data: 2024-04-09T21:58:21.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T21:58:21.124 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:21.124 DEBUG:boto:Port: 8001 2024-04-09T21:58:21.124 DEBUG:boto:Params: {} 2024-04-09T21:58:21.124 DEBUG:boto:Token: None 2024-04-09T21:58:21.124 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a2 2024-04-09T21:58:21.124 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TEiKK1jsAtcxRC87PHh6J28GQUQ= 2024-04-09T21:58:21.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TEiKK1jsAtcxRC87PHh6J28GQUQ='} 2024-04-09T21:58:21.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qCoBxRAr1Scbxlu8nti6HCONh5jOYm8'), ('x-amz-request-id', 'tx00000c932dcc9afb24d9c-006615b9fd-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.167 DEBUG:rgw_multi.tests:version3 id=qCoBxRAr1Scbxlu8nti6HCONh5jOYm8 2024-04-09T21:58:21.167 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.167 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:58:21.167 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=qCoBxRAr1Scbxlu8nti6HCONh5jOYm8 2024-04-09T21:58:21.167 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=qCoBxRAr1Scbxlu8nti6HCONh5jOYm8 2024-04-09T21:58:21.167 DEBUG:boto:Method: DELETE 2024-04-09T21:58:21.168 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=qCoBxRAr1Scbxlu8nti6HCONh5jOYm8 2024-04-09T21:58:21.168 DEBUG:boto:Data: 2024-04-09T21:58:21.168 DEBUG:boto:Headers: {} 2024-04-09T21:58:21.168 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:58:21.168 DEBUG:boto:Port: 8001 2024-04-09T21:58:21.168 DEBUG:boto:Params: {} 2024-04-09T21:58:21.168 DEBUG:boto:Token: None 2024-04-09T21:58:21.168 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 21:58:21 GMT /xryqqk-14/obj-a2?versionId=qCoBxRAr1Scbxlu8nti6HCONh5jOYm8 2024-04-09T21:58:21.169 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qRFVU7/ET3B1hzpBo8LVEACqg4c= 2024-04-09T21:58:21.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:58:21 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qRFVU7/ET3B1hzpBo8LVEACqg4c=', 'Content-Length': '0'} 2024-04-09T21:58:21.177 DEBUG:boto:Response headers: [('x-amz-version-id', 'qCoBxRAr1Scbxlu8nti6HCONh5jOYm8'), ('x-amz-request-id', 'tx000008203fd3e9fc08078-006615b9fd-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:58:21.177 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-13 2024-04-09T21:58:21.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:58:21.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:58:21.178 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:58:21.234 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:58:21.235 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:58:21.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:21.270+0000 7fbe995cb780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:21.271 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , , , ] 2024-04-09T21:58:21.271 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.22.6, ] 2024-04-09T21:58:26.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:26.271+0000 7fbe995cb780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:26.273 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , , , ] 2024-04-09T21:58:26.273 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.22.6, ] 2024-04-09T21:58:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:31.272+0000 7fbe995cb780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:31.274 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , , , ] 2024-04-09T21:58:31.274 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.22.6, ] 2024-04-09T21:58:36.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:36.274+0000 7fbe995cb780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:36.275 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , , , ] 2024-04-09T21:58:36.275 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.22.6, ] 2024-04-09T21:58:41.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:41.275+0000 7fbe995cb780 1 bucket sync caught up with source: 2024-04-09T21:58:41.277 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000010.22.6, ] 2024-04-09T21:58:41.277 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , , 00000000010.22.6, ] 2024-04-09T21:58:41.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:41.275+0000 7fbe995cb780 0 bucket checkpoint complete 2024-04-09T21:58:41.294 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-13 2024-04-09T21:58:41.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:58:41.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:58:41.295 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:58:41.357 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:58:41.357 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:58:41.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:41.394+0000 7f0c06d0e780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:41.396 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000010.38.6, , ] 2024-04-09T21:58:41.396 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000010.38.6, 00000000002.11.11, ] 2024-04-09T21:58:46.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:46.395+0000 7f0c06d0e780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:46.397 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000010.38.6, , ] 2024-04-09T21:58:46.397 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000010.38.6, 00000000002.11.11, ] 2024-04-09T21:58:51.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:51.397+0000 7f0c06d0e780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:51.398 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000010.38.6, , ] 2024-04-09T21:58:51.398 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000010.38.6, 00000000002.11.11, ] 2024-04-09T21:58:56.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:58:56.398+0000 7f0c06d0e780 1 waiting for incremental sync to catch up: 2024-04-09T21:58:56.400 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000010.38.6, , ] 2024-04-09T21:58:56.400 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000010.38.6, 00000000002.11.11, ] 2024-04-09T21:59:01.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:01.399+0000 7f0c06d0e780 1 bucket sync caught up with source: 2024-04-09T21:59:01.401 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000010.38.6, 00000000002.11.11, ] 2024-04-09T21:59:01.401 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000010.38.6, 00000000002.11.11, ] 2024-04-09T21:59:01.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:01.399+0000 7f0c06d0e780 0 bucket checkpoint complete 2024-04-09T21:59:01.418 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-13 zones={a2, a1} 2024-04-09T21:59:01.418 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.418 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.418 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.418 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:59:01.418 DEBUG:boto:Data: 2024-04-09T21:59:01.418 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.419 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.419 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.419 DEBUG:boto:Params: {} 2024-04-09T21:59:01.419 DEBUG:boto:Token: None 2024-04-09T21:59:01.419 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.419 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u7R14OlDvIrL1yrLTTlfjOdyVg4= 2024-04-09T21:59:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u7R14OlDvIrL1yrLTTlfjOdyVg4=', 'Content-Length': '0'} 2024-04-09T21:59:01.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c4008040e9e146f-006615ba25-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.423 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.423 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.423 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.423 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:59:01.423 DEBUG:boto:Data: 2024-04-09T21:59:01.423 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.423 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.423 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.423 DEBUG:boto:Params: {} 2024-04-09T21:59:01.423 DEBUG:boto:Token: None 2024-04-09T21:59:01.423 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.424 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u7R14OlDvIrL1yrLTTlfjOdyVg4= 2024-04-09T21:59:01.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u7R14OlDvIrL1yrLTTlfjOdyVg4=', 'Content-Length': '0'} 2024-04-09T21:59:01.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f21aa961a066287-006615ba25-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:59:01.426 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.427 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.427 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.427 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.427 DEBUG:boto:Method: GET 2024-04-09T21:59:01.427 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.427 DEBUG:boto:Data: 2024-04-09T21:59:01.427 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.427 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.427 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.427 DEBUG:boto:Params: {} 2024-04-09T21:59:01.427 DEBUG:boto:Token: None 2024-04-09T21:59:01.428 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/?versions 2024-04-09T21:59:01.428 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I= 2024-04-09T21:59:01.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I=', 'Content-Length': '0'} 2024-04-09T21:59:01.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cabcaf04281d0399-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.431 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.432 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:01.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:01.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:01.432 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:59:01.432 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.432 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.432 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.432 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.432 DEBUG:boto:Method: GET 2024-04-09T21:59:01.433 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.433 DEBUG:boto:Data: 2024-04-09T21:59:01.433 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.433 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.433 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.433 DEBUG:boto:Params: {} 2024-04-09T21:59:01.433 DEBUG:boto:Token: None 2024-04-09T21:59:01.433 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/?versions 2024-04-09T21:59:01.433 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I= 2024-04-09T21:59:01.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I=', 'Content-Length': '0'} 2024-04-09T21:59:01.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096daa8347f49e2f4-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.436 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.436 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:01.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:01.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:01.437 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.437 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.437 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.437 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.437 DEBUG:boto:Method: GET 2024-04-09T21:59:01.437 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.437 DEBUG:boto:Data: 2024-04-09T21:59:01.437 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.437 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.438 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.438 DEBUG:boto:Params: {} 2024-04-09T21:59:01.438 DEBUG:boto:Token: None 2024-04-09T21:59:01.438 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/?versions 2024-04-09T21:59:01.438 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I= 2024-04-09T21:59:01.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I=', 'Content-Length': '0'} 2024-04-09T21:59:01.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d87ca5865cc169c-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.441 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.442 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.442 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.442 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.442 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.442 DEBUG:boto:Method: GET 2024-04-09T21:59:01.442 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:01.442 DEBUG:boto:Data: 2024-04-09T21:59:01.442 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.442 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.442 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.442 DEBUG:boto:Params: {} 2024-04-09T21:59:01.443 DEBUG:boto:Token: None 2024-04-09T21:59:01.443 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/?versions 2024-04-09T21:59:01.443 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I= 2024-04-09T21:59:01.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GeVRc8gXS8Rcn58tftHhfZbci0I=', 'Content-Length': '0'} 2024-04-09T21:59:01.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e84c0e0811365d1-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.446 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:01.446 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.447 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.447 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.447 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.447 DEBUG:boto:Method: GET 2024-04-09T21:59:01.447 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.447 DEBUG:boto:Data: 2024-04-09T21:59:01.447 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.447 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.447 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.447 DEBUG:boto:Params: {} 2024-04-09T21:59:01.448 DEBUG:boto:Token: None 2024-04-09T21:59:01.448 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.448 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ= 2024-04-09T21:59:01.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c93f69179f426b5-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.450 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.450 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.450 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.450 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.451 DEBUG:boto:Method: GET 2024-04-09T21:59:01.451 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.451 DEBUG:boto:Data: 2024-04-09T21:59:01.451 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.451 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.451 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.451 DEBUG:boto:Params: {} 2024-04-09T21:59:01.451 DEBUG:boto:Token: None 2024-04-09T21:59:01.451 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.451 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ= 2024-04-09T21:59:01.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000982888aab9a96a0c-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.453 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.453 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.454 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.454 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.454 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.454 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.454 DEBUG:boto:Data: 2024-04-09T21:59:01.454 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.454 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.454 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.454 DEBUG:boto:Params: {} 2024-04-09T21:59:01.454 DEBUG:boto:Token: None 2024-04-09T21:59:01.455 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.455 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y6NJpp/VJBpTgcDbddiVQpNgvOQ= 2024-04-09T21:59:01.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y6NJpp/VJBpTgcDbddiVQpNgvOQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.456 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000850f74ebdf4ea299-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.457 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.457 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.457 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.457 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.457 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.457 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.457 DEBUG:boto:Data: 2024-04-09T21:59:01.458 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.458 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.458 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.458 DEBUG:boto:Params: {} 2024-04-09T21:59:01.458 DEBUG:boto:Token: None 2024-04-09T21:59:01.458 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.458 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y6NJpp/VJBpTgcDbddiVQpNgvOQ= 2024-04-09T21:59:01.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y6NJpp/VJBpTgcDbddiVQpNgvOQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003dd481361d8cfff9-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:01.460 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.461 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.461 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.461 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.461 DEBUG:boto:Method: GET 2024-04-09T21:59:01.461 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.462 DEBUG:boto:Data: 2024-04-09T21:59:01.462 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.462 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.462 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.462 DEBUG:boto:Params: {} 2024-04-09T21:59:01.463 DEBUG:boto:Token: None 2024-04-09T21:59:01.463 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.463 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ= 2024-04-09T21:59:01.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c23913bc599c400d-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.465 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.466 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.466 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.466 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.466 DEBUG:boto:Method: GET 2024-04-09T21:59:01.466 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.467 DEBUG:boto:Data: 2024-04-09T21:59:01.467 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.467 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.467 DEBUG:boto:Params: {} 2024-04-09T21:59:01.468 DEBUG:boto:Token: None 2024-04-09T21:59:01.468 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ= 2024-04-09T21:59:01.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000de23a4cbf3a57b50-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.471 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.471 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.471 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.471 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.472 DEBUG:boto:Method: GET 2024-04-09T21:59:01.472 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.472 DEBUG:boto:Data: 2024-04-09T21:59:01.472 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.472 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.473 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.473 DEBUG:boto:Params: {} 2024-04-09T21:59:01.473 DEBUG:boto:Token: None 2024-04-09T21:59:01.473 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.473 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060c5715102132ec-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.477 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.477 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.477 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.478 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.478 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.478 DEBUG:boto:Method: GET 2024-04-09T21:59:01.478 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.478 DEBUG:boto:Data: 2024-04-09T21:59:01.478 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.478 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.478 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.478 DEBUG:boto:Params: {} 2024-04-09T21:59:01.479 DEBUG:boto:Token: None 2024-04-09T21:59:01.479 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.479 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d61ea98c15a31296-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.526 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.527 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.527 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.527 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.527 DEBUG:boto:Path: /xryqqk-13/dummy 2024-04-09T21:59:01.527 DEBUG:boto:Data: 2024-04-09T21:59:01.527 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.527 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.527 DEBUG:boto:Params: {} 2024-04-09T21:59:01.528 DEBUG:boto:Token: None 2024-04-09T21:59:01.528 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy 2024-04-09T21:59:01.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c/VL0SRv99VXTFAi7Yhsx58hPEM= 2024-04-09T21:59:01.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c/VL0SRv99VXTFAi7Yhsx58hPEM=', 'Content-Length': '0'} 2024-04-09T21:59:01.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fdee7d6df1236266-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.531 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.531 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.531 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.531 DEBUG:boto:Path: /xryqqk-13/dummy 2024-04-09T21:59:01.531 DEBUG:boto:Data: 2024-04-09T21:59:01.531 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.531 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.531 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.532 DEBUG:boto:Params: {} 2024-04-09T21:59:01.532 DEBUG:boto:Token: None 2024-04-09T21:59:01.532 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy 2024-04-09T21:59:01.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c/VL0SRv99VXTFAi7Yhsx58hPEM= 2024-04-09T21:59:01.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c/VL0SRv99VXTFAi7Yhsx58hPEM=', 'Content-Length': '0'} 2024-04-09T21:59:01.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006916921eed4dcdff-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:01.535 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.535 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.535 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.535 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.535 DEBUG:boto:Method: GET 2024-04-09T21:59:01.535 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.536 DEBUG:boto:Data: 2024-04-09T21:59:01.536 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.536 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.536 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.536 DEBUG:boto:Params: {} 2024-04-09T21:59:01.536 DEBUG:boto:Token: None 2024-04-09T21:59:01.536 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ= 2024-04-09T21:59:01.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023720d609bd0e70c-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.538 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:01.539 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:01.539 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.539 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.539 DEBUG:boto:Method: GET 2024-04-09T21:59:01.539 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.539 DEBUG:boto:Data: 2024-04-09T21:59:01.539 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.539 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.539 DEBUG:boto:Params: {} 2024-04-09T21:59:01.540 DEBUG:boto:Token: None 2024-04-09T21:59:01.540 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:01.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ= 2024-04-09T21:59:01.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r54vychgRnZHS0EnXPTelIvGyJQ=', 'Content-Length': '0'} 2024-04-09T21:59:01.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002fed1e07b0e9391f-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.542 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.542 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.542 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.542 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.542 DEBUG:boto:Method: GET 2024-04-09T21:59:01.542 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.542 DEBUG:boto:Data: 2024-04-09T21:59:01.543 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.543 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.543 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.543 DEBUG:boto:Params: {} 2024-04-09T21:59:01.543 DEBUG:boto:Token: None 2024-04-09T21:59:01.543 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.543 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326a0dfe57d1fe13-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.590 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.591 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.591 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.591 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.591 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.591 DEBUG:boto:Method: GET 2024-04-09T21:59:01.591 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:01.592 DEBUG:boto:Data: 2024-04-09T21:59:01.592 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.592 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.592 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.592 DEBUG:boto:Params: {} 2024-04-09T21:59:01.592 DEBUG:boto:Token: None 2024-04-09T21:59:01.592 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df794730a8801ce1-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.595 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:01.596 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.596 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.596 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.596 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.596 DEBUG:boto:Method: GET 2024-04-09T21:59:01.596 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.596 DEBUG:boto:Data: 2024-04-09T21:59:01.596 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.597 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.597 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.597 DEBUG:boto:Params: {} 2024-04-09T21:59:01.597 DEBUG:boto:Token: None 2024-04-09T21:59:01.597 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.597 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk= 2024-04-09T21:59:01.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk=', 'Content-Length': '0'} 2024-04-09T21:59:01.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f4a9fd39b5bbef8-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.599 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.599 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.600 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.600 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.600 DEBUG:boto:Method: GET 2024-04-09T21:59:01.600 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.600 DEBUG:boto:Data: 2024-04-09T21:59:01.600 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.600 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.600 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.600 DEBUG:boto:Params: {} 2024-04-09T21:59:01.600 DEBUG:boto:Token: None 2024-04-09T21:59:01.600 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.601 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk= 2024-04-09T21:59:01.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk=', 'Content-Length': '0'} 2024-04-09T21:59:01.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008096677b227029fe-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.650 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.651 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.651 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.651 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.651 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.651 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.651 DEBUG:boto:Data: 2024-04-09T21:59:01.651 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.651 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.651 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.651 DEBUG:boto:Params: {} 2024-04-09T21:59:01.651 DEBUG:boto:Token: None 2024-04-09T21:59:01.652 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.652 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:P7YXuk4RyacMFf4otI2WhKX19uM= 2024-04-09T21:59:01.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:P7YXuk4RyacMFf4otI2WhKX19uM=', 'Content-Length': '0'} 2024-04-09T21:59:01.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011c7c27adc437c71-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.654 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.654 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.654 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.654 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.654 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.654 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.654 DEBUG:boto:Data: 2024-04-09T21:59:01.654 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.654 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.655 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.655 DEBUG:boto:Params: {} 2024-04-09T21:59:01.655 DEBUG:boto:Token: None 2024-04-09T21:59:01.655 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.655 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:P7YXuk4RyacMFf4otI2WhKX19uM= 2024-04-09T21:59:01.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:P7YXuk4RyacMFf4otI2WhKX19uM=', 'Content-Length': '0'} 2024-04-09T21:59:01.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df14f37eb4abd506-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:01.657 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.657 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.658 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.658 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.658 DEBUG:boto:Method: GET 2024-04-09T21:59:01.658 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.658 DEBUG:boto:Data: 2024-04-09T21:59:01.658 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.658 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.658 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.658 DEBUG:boto:Params: {} 2024-04-09T21:59:01.658 DEBUG:boto:Token: None 2024-04-09T21:59:01.659 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.659 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk= 2024-04-09T21:59:01.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk=', 'Content-Length': '0'} 2024-04-09T21:59:01.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000607d8ce1a363c8e6-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.702 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.702 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.702 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.703 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.703 DEBUG:boto:Method: GET 2024-04-09T21:59:01.703 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.703 DEBUG:boto:Data: 2024-04-09T21:59:01.703 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.703 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.703 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.703 DEBUG:boto:Params: {} 2024-04-09T21:59:01.703 DEBUG:boto:Token: None 2024-04-09T21:59:01.703 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.704 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk= 2024-04-09T21:59:01.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk=', 'Content-Length': '0'} 2024-04-09T21:59:01.705 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f2ad49b8d2dbb20-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.706 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.706 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.706 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.706 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.706 DEBUG:boto:Method: GET 2024-04-09T21:59:01.706 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.706 DEBUG:boto:Data: 2024-04-09T21:59:01.706 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.706 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.707 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.707 DEBUG:boto:Params: {} 2024-04-09T21:59:01.707 DEBUG:boto:Token: None 2024-04-09T21:59:01.707 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.707 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f75d6bd04ffccb-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.710 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.710 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.711 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.711 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.711 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.711 DEBUG:boto:Method: GET 2024-04-09T21:59:01.711 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.711 DEBUG:boto:Data: 2024-04-09T21:59:01.711 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.711 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.711 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.711 DEBUG:boto:Params: {} 2024-04-09T21:59:01.711 DEBUG:boto:Token: None 2024-04-09T21:59:01.712 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.712 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc83cc7e8fc13a55-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.714 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.715 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.715 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.715 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.715 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:59:01.715 DEBUG:boto:Data: 2024-04-09T21:59:01.715 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.715 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.715 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.715 DEBUG:boto:Params: {} 2024-04-09T21:59:01.715 DEBUG:boto:Token: None 2024-04-09T21:59:01.716 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1 2024-04-09T21:59:01.716 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KRPSISwpse4pDOK7aJ6neQgnJJ4= 2024-04-09T21:59:01.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KRPSISwpse4pDOK7aJ6neQgnJJ4=', 'Content-Length': '0'} 2024-04-09T21:59:01.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3a00aa2d1b84826-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.718 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.718 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.718 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.718 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:59:01.719 DEBUG:boto:Data: 2024-04-09T21:59:01.719 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.719 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.719 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.719 DEBUG:boto:Params: {} 2024-04-09T21:59:01.719 DEBUG:boto:Token: None 2024-04-09T21:59:01.719 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1 2024-04-09T21:59:01.719 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KRPSISwpse4pDOK7aJ6neQgnJJ4= 2024-04-09T21:59:01.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KRPSISwpse4pDOK7aJ6neQgnJJ4=', 'Content-Length': '0'} 2024-04-09T21:59:01.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ffbaa2606d35246-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:01.721 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.721 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.722 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.722 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.722 DEBUG:boto:Method: GET 2024-04-09T21:59:01.722 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.722 DEBUG:boto:Data: 2024-04-09T21:59:01.722 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.722 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.722 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.722 DEBUG:boto:Params: {} 2024-04-09T21:59:01.722 DEBUG:boto:Token: None 2024-04-09T21:59:01.723 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.723 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk= 2024-04-09T21:59:01.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk=', 'Content-Length': '0'} 2024-04-09T21:59:01.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b129e7da27c455ed-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.766 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.766 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:01.767 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.767 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.767 DEBUG:boto:Method: GET 2024-04-09T21:59:01.767 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.767 DEBUG:boto:Data: 2024-04-09T21:59:01.767 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.767 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.767 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.767 DEBUG:boto:Params: {} 2024-04-09T21:59:01.767 DEBUG:boto:Token: None 2024-04-09T21:59:01.767 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:01.768 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk= 2024-04-09T21:59:01.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HYOjyRv5B8SQyySGINbBeaLUglk=', 'Content-Length': '0'} 2024-04-09T21:59:01.769 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efab2e29601e5c78-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.770 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.770 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.770 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.770 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.770 DEBUG:boto:Method: GET 2024-04-09T21:59:01.770 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.770 DEBUG:boto:Data: 2024-04-09T21:59:01.770 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.770 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.770 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.771 DEBUG:boto:Params: {} 2024-04-09T21:59:01.771 DEBUG:boto:Token: None 2024-04-09T21:59:01.771 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.771 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650e8c3a27f7e6aa-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.814 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.815 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.815 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.815 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.815 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.815 DEBUG:boto:Method: GET 2024-04-09T21:59:01.815 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:01.815 DEBUG:boto:Data: 2024-04-09T21:59:01.815 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.815 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.815 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.815 DEBUG:boto:Params: {} 2024-04-09T21:59:01.816 DEBUG:boto:Token: None 2024-04-09T21:59:01.816 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.816 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2f5e6588ff6a3d6-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.819 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:01.819 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.819 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.819 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.820 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.820 DEBUG:boto:Method: GET 2024-04-09T21:59:01.820 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.820 DEBUG:boto:Data: 2024-04-09T21:59:01.820 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.820 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.820 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.820 DEBUG:boto:Params: {} 2024-04-09T21:59:01.820 DEBUG:boto:Token: None 2024-04-09T21:59:01.820 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.820 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y= 2024-04-09T21:59:01.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y=', 'Content-Length': '0'} 2024-04-09T21:59:01.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fa14ba18e53d2dd-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.823 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.823 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.823 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.823 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.823 DEBUG:boto:Method: GET 2024-04-09T21:59:01.823 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.823 DEBUG:boto:Data: 2024-04-09T21:59:01.823 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.823 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.823 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.823 DEBUG:boto:Params: {} 2024-04-09T21:59:01.824 DEBUG:boto:Token: None 2024-04-09T21:59:01.824 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y= 2024-04-09T21:59:01.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y=', 'Content-Length': '0'} 2024-04-09T21:59:01.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f54e34fdedbd192d-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.866 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.866 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.867 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.867 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.867 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.867 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.867 DEBUG:boto:Data: 2024-04-09T21:59:01.867 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.867 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.867 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.867 DEBUG:boto:Params: {} 2024-04-09T21:59:01.867 DEBUG:boto:Token: None 2024-04-09T21:59:01.867 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.868 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7vJgwz0d3UuA+QV4BDcPWgqzfPw= 2024-04-09T21:59:01.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7vJgwz0d3UuA+QV4BDcPWgqzfPw=', 'Content-Length': '0'} 2024-04-09T21:59:01.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000154d8857d52ec8f0-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.870 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.870 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.870 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.870 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.870 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.870 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.870 DEBUG:boto:Data: 2024-04-09T21:59:01.871 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.871 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.871 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.871 DEBUG:boto:Params: {} 2024-04-09T21:59:01.871 DEBUG:boto:Token: None 2024-04-09T21:59:01.871 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.871 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7vJgwz0d3UuA+QV4BDcPWgqzfPw= 2024-04-09T21:59:01.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7vJgwz0d3UuA+QV4BDcPWgqzfPw=', 'Content-Length': '0'} 2024-04-09T21:59:01.873 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e70d1f788100128-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:01.873 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.873 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.873 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.874 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.874 DEBUG:boto:Method: GET 2024-04-09T21:59:01.874 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.874 DEBUG:boto:Data: 2024-04-09T21:59:01.874 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.874 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.874 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.874 DEBUG:boto:Params: {} 2024-04-09T21:59:01.874 DEBUG:boto:Token: None 2024-04-09T21:59:01.874 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.875 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y= 2024-04-09T21:59:01.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y=', 'Content-Length': '0'} 2024-04-09T21:59:01.876 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad1a350ef2bcc215-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.877 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.877 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.877 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.877 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.877 DEBUG:boto:Method: GET 2024-04-09T21:59:01.877 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.877 DEBUG:boto:Data: 2024-04-09T21:59:01.877 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.877 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.877 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.877 DEBUG:boto:Params: {} 2024-04-09T21:59:01.877 DEBUG:boto:Token: None 2024-04-09T21:59:01.878 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.878 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y= 2024-04-09T21:59:01.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y=', 'Content-Length': '0'} 2024-04-09T21:59:01.879 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000806ea7c9bfdaec06-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.927 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.927 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.927 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.927 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.927 DEBUG:boto:Method: GET 2024-04-09T21:59:01.927 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.927 DEBUG:boto:Data: 2024-04-09T21:59:01.927 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.927 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.927 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.927 DEBUG:boto:Params: {} 2024-04-09T21:59:01.928 DEBUG:boto:Token: None 2024-04-09T21:59:01.928 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.928 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c202843d5e68c81f-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.931 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.931 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.931 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.931 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.931 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.932 DEBUG:boto:Method: GET 2024-04-09T21:59:01.932 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.932 DEBUG:boto:Data: 2024-04-09T21:59:01.932 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.932 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.932 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.932 DEBUG:boto:Params: {} 2024-04-09T21:59:01.932 DEBUG:boto:Token: None 2024-04-09T21:59:01.932 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.932 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d08d20048030304-006615ba25-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.935 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.936 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.936 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.936 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.936 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:59:01.936 DEBUG:boto:Data: 2024-04-09T21:59:01.936 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.936 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.936 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.936 DEBUG:boto:Params: {} 2024-04-09T21:59:01.936 DEBUG:boto:Token: None 2024-04-09T21:59:01.937 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2 2024-04-09T21:59:01.937 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ytrM8D3HQ3ZFndBNJNnGeudvQNs= 2024-04-09T21:59:01.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ytrM8D3HQ3ZFndBNJNnGeudvQNs=', 'Content-Length': '0'} 2024-04-09T21:59:01.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a293b0f7924281fe-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.939 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.939 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.939 DEBUG:boto:Method: HEAD 2024-04-09T21:59:01.940 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:59:01.940 DEBUG:boto:Data: 2024-04-09T21:59:01.940 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.940 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.940 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.940 DEBUG:boto:Params: {} 2024-04-09T21:59:01.940 DEBUG:boto:Token: None 2024-04-09T21:59:01.940 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2 2024-04-09T21:59:01.940 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ytrM8D3HQ3ZFndBNJNnGeudvQNs= 2024-04-09T21:59:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ytrM8D3HQ3ZFndBNJNnGeudvQNs=', 'Content-Length': '0'} 2024-04-09T21:59:01.942 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd5ba9bf245edf47-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:01.945 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.945 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.945 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.945 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.946 DEBUG:boto:Method: GET 2024-04-09T21:59:01.946 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.946 DEBUG:boto:Data: 2024-04-09T21:59:01.946 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.946 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.946 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.946 DEBUG:boto:Params: {} 2024-04-09T21:59:01.946 DEBUG:boto:Token: None 2024-04-09T21:59:01.946 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.947 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y= 2024-04-09T21:59:01.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y=', 'Content-Length': '0'} 2024-04-09T21:59:01.949 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009468535848d0c4da-006615ba25-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.990 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.991 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:01.991 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.991 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.991 DEBUG:boto:Method: GET 2024-04-09T21:59:01.991 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.991 DEBUG:boto:Data: 2024-04-09T21:59:01.991 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.991 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:01.991 DEBUG:boto:Port: 8000 2024-04-09T21:59:01.991 DEBUG:boto:Params: {} 2024-04-09T21:59:01.992 DEBUG:boto:Token: None 2024-04-09T21:59:01.992 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:01.992 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y= 2024-04-09T21:59:01.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h4vyC8teUejwEuTII2gXuW2Gy7Y=', 'Content-Length': '0'} 2024-04-09T21:59:01.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037a478303f0c4be4-006615ba25-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.994 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.994 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.995 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.995 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.995 DEBUG:boto:Method: GET 2024-04-09T21:59:01.995 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.995 DEBUG:boto:Data: 2024-04-09T21:59:01.995 DEBUG:boto:Headers: {} 2024-04-09T21:59:01.995 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:01.995 DEBUG:boto:Port: 8001 2024-04-09T21:59:01.995 DEBUG:boto:Params: {} 2024-04-09T21:59:01.995 DEBUG:boto:Token: None 2024-04-09T21:59:01.996 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:01 GMT /xryqqk-13/ 2024-04-09T21:59:01.996 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA= 2024-04-09T21:59:01.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8IJK8jtoZv8RQoJG0mQEZpUtNUA=', 'Content-Length': '0'} 2024-04-09T21:59:01.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0335dd057dde3c2-006615ba25-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:01.999 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:01.999 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:01.999 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:01.999 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.999 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:01.999 DEBUG:boto:Method: GET 2024-04-09T21:59:02.000 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:02.000 DEBUG:boto:Data: 2024-04-09T21:59:02.000 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.000 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.000 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.000 DEBUG:boto:Params: {} 2024-04-09T21:59:02.000 DEBUG:boto:Token: None 2024-04-09T21:59:02.000 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-13/ 2024-04-09T21:59:02.000 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iUDGPGS9XbhMLBF0+zhtVXlke8A= 2024-04-09T21:59:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iUDGPGS9XbhMLBF0+zhtVXlke8A=', 'Content-Length': '0'} 2024-04-09T21:59:02.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019590a6ce4132628-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.003 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-13 zones={a2, a1} 2024-04-09T21:59:02.003 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-14 2024-04-09T21:59:02.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:02.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:02.004 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:02.068 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:02.069 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:02.109+0000 7fa44fa4d780 1 bucket sync caught up with source: 2024-04-09T21:59:02.110 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.9.11, , , 00000000002.5.11, 00000000010.23.6, ] 2024-04-09T21:59:02.111 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.9.11, , , 00000000002.5.11, 00000000010.23.6, ] 2024-04-09T21:59:02.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:02.109+0000 7fa44fa4d780 0 bucket checkpoint complete 2024-04-09T21:59:02.124 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-14 2024-04-09T21:59:02.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:02.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:02.124 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:02.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:02.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:02.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:02.215+0000 7f27318c0780 1 bucket sync caught up with source: 2024-04-09T21:59:02.216 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.18.10, , , 00000000010.24.6, 00000000002.13.11, ] 2024-04-09T21:59:02.216 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000010.24.6, 00000000002.13.11, ] 2024-04-09T21:59:02.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:02.215+0000 7f27318c0780 0 bucket checkpoint complete 2024-04-09T21:59:02.229 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-14 zones={a2, a1} 2024-04-09T21:59:02.229 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.229 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.229 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.229 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:59:02.229 DEBUG:boto:Data: 2024-04-09T21:59:02.229 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.230 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.230 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.230 DEBUG:boto:Params: {} 2024-04-09T21:59:02.230 DEBUG:boto:Token: None 2024-04-09T21:59:02.230 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.230 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU= 2024-04-09T21:59:02.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU=', 'Content-Length': '0'} 2024-04-09T21:59:02.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000240fed1d12f109ab-006615ba26-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.233 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.233 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.233 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.233 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:59:02.233 DEBUG:boto:Data: 2024-04-09T21:59:02.233 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.233 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.233 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.233 DEBUG:boto:Params: {} 2024-04-09T21:59:02.234 DEBUG:boto:Token: None 2024-04-09T21:59:02.234 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.234 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU= 2024-04-09T21:59:02.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU=', 'Content-Length': '0'} 2024-04-09T21:59:02.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ba7e1bcc70b0397-006615ba26-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.236 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:59:02.236 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.236 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.236 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.236 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.236 DEBUG:boto:Method: GET 2024-04-09T21:59:02.237 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.237 DEBUG:boto:Data: 2024-04-09T21:59:02.237 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.237 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.237 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.237 DEBUG:boto:Params: {} 2024-04-09T21:59:02.237 DEBUG:boto:Token: None 2024-04-09T21:59:02.237 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/?versions 2024-04-09T21:59:02.237 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20= 2024-04-09T21:59:02.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20=', 'Content-Length': '0'} 2024-04-09T21:59:02.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f675ea1194f780d-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.240 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.240 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:02.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:02.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:02.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:59:02.240 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.241 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.241 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.241 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.241 DEBUG:boto:Method: GET 2024-04-09T21:59:02.241 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.241 DEBUG:boto:Data: 2024-04-09T21:59:02.241 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.241 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.241 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.241 DEBUG:boto:Params: {} 2024-04-09T21:59:02.241 DEBUG:boto:Token: None 2024-04-09T21:59:02.242 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/?versions 2024-04-09T21:59:02.242 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20= 2024-04-09T21:59:02.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20=', 'Content-Length': '0'} 2024-04-09T21:59:02.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f99535f37fd712c-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.244 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.245 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:02.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:02.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:02.245 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.245 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.245 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.246 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.246 DEBUG:boto:Method: GET 2024-04-09T21:59:02.246 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.246 DEBUG:boto:Data: 2024-04-09T21:59:02.246 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.246 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.246 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.246 DEBUG:boto:Params: {} 2024-04-09T21:59:02.246 DEBUG:boto:Token: None 2024-04-09T21:59:02.246 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/?versions 2024-04-09T21:59:02.247 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20= 2024-04-09T21:59:02.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20=', 'Content-Length': '0'} 2024-04-09T21:59:02.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006998d6b7bc0fac19-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.249 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.249 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.249 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.249 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.249 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.250 DEBUG:boto:Method: GET 2024-04-09T21:59:02.250 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:02.250 DEBUG:boto:Data: 2024-04-09T21:59:02.250 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.250 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.250 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.250 DEBUG:boto:Params: {} 2024-04-09T21:59:02.250 DEBUG:boto:Token: None 2024-04-09T21:59:02.250 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/?versions 2024-04-09T21:59:02.251 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20= 2024-04-09T21:59:02.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20=', 'Content-Length': '0'} 2024-04-09T21:59:02.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196b09231e9e2a7c-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.253 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:02.254 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.254 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.254 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.254 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.254 DEBUG:boto:Method: GET 2024-04-09T21:59:02.254 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.254 DEBUG:boto:Data: 2024-04-09T21:59:02.254 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.254 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.254 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.255 DEBUG:boto:Params: {} 2024-04-09T21:59:02.255 DEBUG:boto:Token: None 2024-04-09T21:59:02.255 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.255 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA= 2024-04-09T21:59:02.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA=', 'Content-Length': '0'} 2024-04-09T21:59:02.256 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009eaed0c75360de7a-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.257 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.257 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.257 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.257 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.257 DEBUG:boto:Method: GET 2024-04-09T21:59:02.257 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.257 DEBUG:boto:Data: 2024-04-09T21:59:02.257 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.258 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.258 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.258 DEBUG:boto:Params: {} 2024-04-09T21:59:02.258 DEBUG:boto:Token: None 2024-04-09T21:59:02.258 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.258 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA= 2024-04-09T21:59:02.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA=', 'Content-Length': '0'} 2024-04-09T21:59:02.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000612ca691702604d4-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.260 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.260 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.260 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.260 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.260 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.260 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.260 DEBUG:boto:Data: 2024-04-09T21:59:02.260 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.260 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.260 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.261 DEBUG:boto:Params: {} 2024-04-09T21:59:02.261 DEBUG:boto:Token: None 2024-04-09T21:59:02.261 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.261 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YIQ1dmJJRH9EzRw8SA4V62/y6ww= 2024-04-09T21:59:02.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YIQ1dmJJRH9EzRw8SA4V62/y6ww=', 'Content-Length': '0'} 2024-04-09T21:59:02.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097c16186d2d0ad3a-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.263 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.263 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.263 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.263 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.263 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.263 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.263 DEBUG:boto:Data: 2024-04-09T21:59:02.264 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.264 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.264 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.264 DEBUG:boto:Params: {} 2024-04-09T21:59:02.264 DEBUG:boto:Token: None 2024-04-09T21:59:02.264 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.264 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YIQ1dmJJRH9EzRw8SA4V62/y6ww= 2024-04-09T21:59:02.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YIQ1dmJJRH9EzRw8SA4V62/y6ww=', 'Content-Length': '0'} 2024-04-09T21:59:02.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000985f1f00385e9f7b-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:02.266 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.266 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.266 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.266 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.266 DEBUG:boto:Method: GET 2024-04-09T21:59:02.267 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.267 DEBUG:boto:Data: 2024-04-09T21:59:02.267 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.267 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.267 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.267 DEBUG:boto:Params: {} 2024-04-09T21:59:02.267 DEBUG:boto:Token: None 2024-04-09T21:59:02.267 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.267 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA= 2024-04-09T21:59:02.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA=', 'Content-Length': '0'} 2024-04-09T21:59:02.269 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c4f76a698b0826f-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.269 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.269 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.269 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.269 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.269 DEBUG:boto:Method: GET 2024-04-09T21:59:02.269 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.269 DEBUG:boto:Data: 2024-04-09T21:59:02.270 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.270 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.270 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.270 DEBUG:boto:Params: {} 2024-04-09T21:59:02.270 DEBUG:boto:Token: None 2024-04-09T21:59:02.270 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.270 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA= 2024-04-09T21:59:02.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA=', 'Content-Length': '0'} 2024-04-09T21:59:02.272 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c33f345cc813d0ff-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.272 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.272 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.272 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.272 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.273 DEBUG:boto:Method: GET 2024-04-09T21:59:02.273 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.273 DEBUG:boto:Data: 2024-04-09T21:59:02.273 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.273 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.273 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.273 DEBUG:boto:Params: {} 2024-04-09T21:59:02.273 DEBUG:boto:Token: None 2024-04-09T21:59:02.273 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.273 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045dfebb548590f96-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.318 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.319 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.319 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.319 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.319 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.319 DEBUG:boto:Method: GET 2024-04-09T21:59:02.319 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.319 DEBUG:boto:Data: 2024-04-09T21:59:02.319 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.319 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.319 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.319 DEBUG:boto:Params: {} 2024-04-09T21:59:02.320 DEBUG:boto:Token: None 2024-04-09T21:59:02.320 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3d9d76d76391a6a-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.322 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.323 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.323 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.323 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.323 DEBUG:boto:Path: /xryqqk-14/dummy 2024-04-09T21:59:02.323 DEBUG:boto:Data: 2024-04-09T21:59:02.323 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.323 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.323 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.323 DEBUG:boto:Params: {} 2024-04-09T21:59:02.323 DEBUG:boto:Token: None 2024-04-09T21:59:02.324 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy 2024-04-09T21:59:02.324 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/s7gl8nsYMWhMGpD2Uj2eheawqM= 2024-04-09T21:59:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/s7gl8nsYMWhMGpD2Uj2eheawqM=', 'Content-Length': '0'} 2024-04-09T21:59:02.326 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dd23556380ec0c94-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.326 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.326 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.326 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.326 DEBUG:boto:Path: /xryqqk-14/dummy 2024-04-09T21:59:02.326 DEBUG:boto:Data: 2024-04-09T21:59:02.326 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.326 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.327 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.327 DEBUG:boto:Params: {} 2024-04-09T21:59:02.327 DEBUG:boto:Token: None 2024-04-09T21:59:02.327 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy 2024-04-09T21:59:02.327 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/s7gl8nsYMWhMGpD2Uj2eheawqM= 2024-04-09T21:59:02.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/s7gl8nsYMWhMGpD2Uj2eheawqM=', 'Content-Length': '0'} 2024-04-09T21:59:02.329 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081b0286da19e08b8-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:02.329 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.330 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.330 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.330 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.330 DEBUG:boto:Method: GET 2024-04-09T21:59:02.330 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.330 DEBUG:boto:Data: 2024-04-09T21:59:02.330 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.330 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.330 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.330 DEBUG:boto:Params: {} 2024-04-09T21:59:02.331 DEBUG:boto:Token: None 2024-04-09T21:59:02.331 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.331 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA= 2024-04-09T21:59:02.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA=', 'Content-Length': '0'} 2024-04-09T21:59:02.332 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e1b3db09c716e6e4-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.332 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:02.333 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:02.333 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.333 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.333 DEBUG:boto:Method: GET 2024-04-09T21:59:02.333 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.333 DEBUG:boto:Data: 2024-04-09T21:59:02.333 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.333 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.333 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.333 DEBUG:boto:Params: {} 2024-04-09T21:59:02.334 DEBUG:boto:Token: None 2024-04-09T21:59:02.334 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:02.334 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA= 2024-04-09T21:59:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vd62Vp8TnoZUysL28ROSAa/UxrA=', 'Content-Length': '0'} 2024-04-09T21:59:02.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ffe0f0f21276693-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.336 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.336 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.336 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.336 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.336 DEBUG:boto:Method: GET 2024-04-09T21:59:02.336 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.336 DEBUG:boto:Data: 2024-04-09T21:59:02.336 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.336 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.336 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.337 DEBUG:boto:Params: {} 2024-04-09T21:59:02.337 DEBUG:boto:Token: None 2024-04-09T21:59:02.337 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.337 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bd0b4ba56d00b97-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.339 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.339 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.339 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.340 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.340 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.340 DEBUG:boto:Method: GET 2024-04-09T21:59:02.340 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:02.340 DEBUG:boto:Data: 2024-04-09T21:59:02.340 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.340 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.340 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.340 DEBUG:boto:Params: {} 2024-04-09T21:59:02.340 DEBUG:boto:Token: None 2024-04-09T21:59:02.340 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.341 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007400ad4a60e0f644-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.343 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:02.343 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.344 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.344 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.344 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.344 DEBUG:boto:Method: GET 2024-04-09T21:59:02.344 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.344 DEBUG:boto:Data: 2024-04-09T21:59:02.344 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.344 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.344 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.344 DEBUG:boto:Params: {} 2024-04-09T21:59:02.344 DEBUG:boto:Token: None 2024-04-09T21:59:02.345 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.345 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o= 2024-04-09T21:59:02.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o=', 'Content-Length': '0'} 2024-04-09T21:59:02.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff647807bc7aa26a-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.347 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.347 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.347 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.347 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.347 DEBUG:boto:Method: GET 2024-04-09T21:59:02.347 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.347 DEBUG:boto:Data: 2024-04-09T21:59:02.347 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.348 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.348 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.348 DEBUG:boto:Params: {} 2024-04-09T21:59:02.348 DEBUG:boto:Token: None 2024-04-09T21:59:02.348 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.348 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o= 2024-04-09T21:59:02.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o=', 'Content-Length': '0'} 2024-04-09T21:59:02.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8f5abc98def85d6-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.350 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.350 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.350 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.350 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.350 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.351 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.351 DEBUG:boto:Data: 2024-04-09T21:59:02.351 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.351 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.351 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.351 DEBUG:boto:Params: {} 2024-04-09T21:59:02.351 DEBUG:boto:Token: None 2024-04-09T21:59:02.351 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.351 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RPdPmhvZyw1wlvSJUAN6rRmaFHo= 2024-04-09T21:59:02.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RPdPmhvZyw1wlvSJUAN6rRmaFHo=', 'Content-Length': '0'} 2024-04-09T21:59:02.353 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a97bb1b4c9b376b5-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.353 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.353 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.353 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.354 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.354 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.354 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.354 DEBUG:boto:Data: 2024-04-09T21:59:02.354 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.354 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.354 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.354 DEBUG:boto:Params: {} 2024-04-09T21:59:02.354 DEBUG:boto:Token: None 2024-04-09T21:59:02.354 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.355 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RPdPmhvZyw1wlvSJUAN6rRmaFHo= 2024-04-09T21:59:02.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RPdPmhvZyw1wlvSJUAN6rRmaFHo=', 'Content-Length': '0'} 2024-04-09T21:59:02.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9adfab4139855dc-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:02.357 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.357 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.357 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.357 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.357 DEBUG:boto:Method: GET 2024-04-09T21:59:02.357 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.357 DEBUG:boto:Data: 2024-04-09T21:59:02.357 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.358 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.358 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.358 DEBUG:boto:Params: {} 2024-04-09T21:59:02.358 DEBUG:boto:Token: None 2024-04-09T21:59:02.358 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.358 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o= 2024-04-09T21:59:02.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o=', 'Content-Length': '0'} 2024-04-09T21:59:02.360 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052ec59f36a5af68a-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.402 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.403 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.403 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.403 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.403 DEBUG:boto:Method: GET 2024-04-09T21:59:02.403 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.403 DEBUG:boto:Data: 2024-04-09T21:59:02.403 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.403 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.403 DEBUG:boto:Params: {} 2024-04-09T21:59:02.404 DEBUG:boto:Token: None 2024-04-09T21:59:02.404 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o= 2024-04-09T21:59:02.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o=', 'Content-Length': '0'} 2024-04-09T21:59:02.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ff3e9294cd36dfc-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.406 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.406 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.406 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.406 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.406 DEBUG:boto:Method: GET 2024-04-09T21:59:02.406 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.407 DEBUG:boto:Data: 2024-04-09T21:59:02.407 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.407 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.407 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.407 DEBUG:boto:Params: {} 2024-04-09T21:59:02.407 DEBUG:boto:Token: None 2024-04-09T21:59:02.407 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.407 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025e91a893df9849-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.410 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.410 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.410 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.410 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.410 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.410 DEBUG:boto:Method: GET 2024-04-09T21:59:02.411 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.411 DEBUG:boto:Data: 2024-04-09T21:59:02.411 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.411 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.411 DEBUG:boto:Params: {} 2024-04-09T21:59:02.411 DEBUG:boto:Token: None 2024-04-09T21:59:02.411 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.411 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abb44c4fe52dbc70-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.413 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.414 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.414 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.414 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.414 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:59:02.414 DEBUG:boto:Data: 2024-04-09T21:59:02.414 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.414 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.414 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.414 DEBUG:boto:Params: {} 2024-04-09T21:59:02.414 DEBUG:boto:Token: None 2024-04-09T21:59:02.415 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1 2024-04-09T21:59:02.415 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:63LS4mFLW3ijhwgSv0c7E5rGUKc= 2024-04-09T21:59:02.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:63LS4mFLW3ijhwgSv0c7E5rGUKc=', 'Content-Length': '0'} 2024-04-09T21:59:02.416 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000500ce2173b6c785a-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.417 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.417 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.417 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.417 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:59:02.417 DEBUG:boto:Data: 2024-04-09T21:59:02.417 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.417 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.417 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.417 DEBUG:boto:Params: {} 2024-04-09T21:59:02.418 DEBUG:boto:Token: None 2024-04-09T21:59:02.418 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1 2024-04-09T21:59:02.418 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:63LS4mFLW3ijhwgSv0c7E5rGUKc= 2024-04-09T21:59:02.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:63LS4mFLW3ijhwgSv0c7E5rGUKc=', 'Content-Length': '0'} 2024-04-09T21:59:02.420 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc7c576ebcb4828-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:02.420 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.420 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.420 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.420 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.420 DEBUG:boto:Method: GET 2024-04-09T21:59:02.421 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.421 DEBUG:boto:Data: 2024-04-09T21:59:02.421 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.421 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.421 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.421 DEBUG:boto:Params: {} 2024-04-09T21:59:02.421 DEBUG:boto:Token: None 2024-04-09T21:59:02.421 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.421 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o= 2024-04-09T21:59:02.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o=', 'Content-Length': '0'} 2024-04-09T21:59:02.423 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a37e8d4b04fafb5-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.466 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.466 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.467 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.467 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.467 DEBUG:boto:Method: GET 2024-04-09T21:59:02.467 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.467 DEBUG:boto:Data: 2024-04-09T21:59:02.467 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.467 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.467 DEBUG:boto:Params: {} 2024-04-09T21:59:02.468 DEBUG:boto:Token: None 2024-04-09T21:59:02.468 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:02.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o= 2024-04-09T21:59:02.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DsdVJli/dFqwK1q+R4lsYq9A71o=', 'Content-Length': '0'} 2024-04-09T21:59:02.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004875620d831c9152-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.470 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.470 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.470 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.470 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.470 DEBUG:boto:Method: GET 2024-04-09T21:59:02.470 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.470 DEBUG:boto:Data: 2024-04-09T21:59:02.470 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.471 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.471 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.471 DEBUG:boto:Params: {} 2024-04-09T21:59:02.471 DEBUG:boto:Token: None 2024-04-09T21:59:02.471 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aadd133e84c34804-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.473 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.474 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.474 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.474 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.474 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.474 DEBUG:boto:Method: GET 2024-04-09T21:59:02.474 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:02.474 DEBUG:boto:Data: 2024-04-09T21:59:02.474 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.474 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.475 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.475 DEBUG:boto:Params: {} 2024-04-09T21:59:02.475 DEBUG:boto:Token: None 2024-04-09T21:59:02.475 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.475 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000856fecd3d88d040c-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.477 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:02.478 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.478 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.478 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.478 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.478 DEBUG:boto:Method: GET 2024-04-09T21:59:02.478 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.478 DEBUG:boto:Data: 2024-04-09T21:59:02.478 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.478 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.478 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.479 DEBUG:boto:Params: {} 2024-04-09T21:59:02.479 DEBUG:boto:Token: None 2024-04-09T21:59:02.479 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.479 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY= 2024-04-09T21:59:02.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY=', 'Content-Length': '0'} 2024-04-09T21:59:02.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000942da336afb8b66f-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.481 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.481 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.481 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.481 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.481 DEBUG:boto:Method: GET 2024-04-09T21:59:02.481 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.481 DEBUG:boto:Data: 2024-04-09T21:59:02.481 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.481 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.482 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.482 DEBUG:boto:Params: {} 2024-04-09T21:59:02.482 DEBUG:boto:Token: None 2024-04-09T21:59:02.482 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.482 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY= 2024-04-09T21:59:02.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY=', 'Content-Length': '0'} 2024-04-09T21:59:02.484 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fde64f0e3f1d92f-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.484 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.484 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.484 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.484 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.484 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.484 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.485 DEBUG:boto:Data: 2024-04-09T21:59:02.485 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.485 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.485 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.485 DEBUG:boto:Params: {} 2024-04-09T21:59:02.485 DEBUG:boto:Token: None 2024-04-09T21:59:02.485 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.485 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bSPxUu6+drymSg9w7me3OiqVHPI= 2024-04-09T21:59:02.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bSPxUu6+drymSg9w7me3OiqVHPI=', 'Content-Length': '0'} 2024-04-09T21:59:02.487 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e5941db5282b981-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.487 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.487 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.487 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.487 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.487 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.487 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.487 DEBUG:boto:Data: 2024-04-09T21:59:02.488 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.488 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.488 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.488 DEBUG:boto:Params: {} 2024-04-09T21:59:02.488 DEBUG:boto:Token: None 2024-04-09T21:59:02.488 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.488 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bSPxUu6+drymSg9w7me3OiqVHPI= 2024-04-09T21:59:02.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bSPxUu6+drymSg9w7me3OiqVHPI=', 'Content-Length': '0'} 2024-04-09T21:59:02.490 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002fee78d762c5e941-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:02.490 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.490 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.490 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.490 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.491 DEBUG:boto:Method: GET 2024-04-09T21:59:02.491 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.491 DEBUG:boto:Data: 2024-04-09T21:59:02.491 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.491 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.491 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.491 DEBUG:boto:Params: {} 2024-04-09T21:59:02.491 DEBUG:boto:Token: None 2024-04-09T21:59:02.491 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.491 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY= 2024-04-09T21:59:02.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY=', 'Content-Length': '0'} 2024-04-09T21:59:02.493 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a95632558eaf0819-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.534 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.534 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.535 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.535 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.535 DEBUG:boto:Method: GET 2024-04-09T21:59:02.535 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.535 DEBUG:boto:Data: 2024-04-09T21:59:02.535 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.535 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.535 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.535 DEBUG:boto:Params: {} 2024-04-09T21:59:02.535 DEBUG:boto:Token: None 2024-04-09T21:59:02.536 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY= 2024-04-09T21:59:02.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY=', 'Content-Length': '0'} 2024-04-09T21:59:02.537 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003990459d7754765e-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.582 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.583 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.583 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.583 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.583 DEBUG:boto:Method: GET 2024-04-09T21:59:02.583 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.583 DEBUG:boto:Data: 2024-04-09T21:59:02.583 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.583 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.583 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.583 DEBUG:boto:Params: {} 2024-04-09T21:59:02.584 DEBUG:boto:Token: None 2024-04-09T21:59:02.584 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.584 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f557438192c895b9-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.586 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.587 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.587 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.587 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.587 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.587 DEBUG:boto:Method: GET 2024-04-09T21:59:02.587 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.587 DEBUG:boto:Data: 2024-04-09T21:59:02.587 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.587 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.588 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.588 DEBUG:boto:Params: {} 2024-04-09T21:59:02.588 DEBUG:boto:Token: None 2024-04-09T21:59:02.588 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.588 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1d8b045cae40d6-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.634 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.635 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.635 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.635 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.635 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:59:02.635 DEBUG:boto:Data: 2024-04-09T21:59:02.635 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.635 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.635 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.635 DEBUG:boto:Params: {} 2024-04-09T21:59:02.635 DEBUG:boto:Token: None 2024-04-09T21:59:02.635 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2 2024-04-09T21:59:02.636 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nGdnGws3qDzgPFqFFTTiZUbTqxo= 2024-04-09T21:59:02.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nGdnGws3qDzgPFqFFTTiZUbTqxo=', 'Content-Length': '0'} 2024-04-09T21:59:02.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f00e73f0c99cf50-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.638 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.638 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.638 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.638 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:59:02.639 DEBUG:boto:Data: 2024-04-09T21:59:02.639 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.639 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.639 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.639 DEBUG:boto:Params: {} 2024-04-09T21:59:02.639 DEBUG:boto:Token: None 2024-04-09T21:59:02.639 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2 2024-04-09T21:59:02.639 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nGdnGws3qDzgPFqFFTTiZUbTqxo= 2024-04-09T21:59:02.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nGdnGws3qDzgPFqFFTTiZUbTqxo=', 'Content-Length': '0'} 2024-04-09T21:59:02.641 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008032c33408925812-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:02.641 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.641 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.642 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.642 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.642 DEBUG:boto:Method: GET 2024-04-09T21:59:02.642 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.642 DEBUG:boto:Data: 2024-04-09T21:59:02.642 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.642 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.642 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.642 DEBUG:boto:Params: {} 2024-04-09T21:59:02.642 DEBUG:boto:Token: None 2024-04-09T21:59:02.642 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.643 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY= 2024-04-09T21:59:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY=', 'Content-Length': '0'} 2024-04-09T21:59:02.644 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ae8583ac44fbd3eb-006615ba26-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.644 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.644 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.644 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.645 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.645 DEBUG:boto:Method: GET 2024-04-09T21:59:02.645 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.645 DEBUG:boto:Data: 2024-04-09T21:59:02.645 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.645 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.645 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.645 DEBUG:boto:Params: {} 2024-04-09T21:59:02.645 DEBUG:boto:Token: None 2024-04-09T21:59:02.645 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:02.645 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY= 2024-04-09T21:59:02.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p5dlmb6I87c8TTwKY2xnLuEEaSY=', 'Content-Length': '0'} 2024-04-09T21:59:02.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b76a60b87463c6b-006615ba26-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.690 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.691 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.691 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.691 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.691 DEBUG:boto:Method: GET 2024-04-09T21:59:02.691 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.691 DEBUG:boto:Data: 2024-04-09T21:59:02.691 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.691 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.691 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.691 DEBUG:boto:Params: {} 2024-04-09T21:59:02.692 DEBUG:boto:Token: None 2024-04-09T21:59:02.692 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.692 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a20a46781913b56-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.694 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.694 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.695 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.695 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.695 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.695 DEBUG:boto:Method: GET 2024-04-09T21:59:02.695 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:02.695 DEBUG:boto:Data: 2024-04-09T21:59:02.695 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.695 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.695 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.695 DEBUG:boto:Params: {} 2024-04-09T21:59:02.695 DEBUG:boto:Token: None 2024-04-09T21:59:02.696 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.696 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8= 2024-04-09T21:59:02.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nOkTpjCyODm650hg4RcZk9lJMI8=', 'Content-Length': '0'} 2024-04-09T21:59:02.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067b49ca76660b189-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.698 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.698 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-14 zones={a2, a1} 2024-04-09T21:59:02.698 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:02.698 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:02.699 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.699 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:59:02.699 DEBUG:boto:Data: 2024-04-09T21:59:02.699 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.699 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.699 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.699 DEBUG:boto:Params: {} 2024-04-09T21:59:02.699 DEBUG:boto:Token: None 2024-04-09T21:59:02.699 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-13/ 2024-04-09T21:59:02.699 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iB46DU8Bsohh2lg9BYjKS0ej2BM= 2024-04-09T21:59:02.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iB46DU8Bsohh2lg9BYjKS0ej2BM=', 'Content-Length': '0'} 2024-04-09T21:59:02.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f6a56d51f2ccc80-006615ba26-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.702 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:02.702 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:02.702 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999&prefix=obj-a1 2024-04-09T21:59:02.702 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999&prefix=obj-a1 2024-04-09T21:59:02.702 DEBUG:boto:Method: GET 2024-04-09T21:59:02.702 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999&prefix=obj-a1 2024-04-09T21:59:02.702 DEBUG:boto:Data: 2024-04-09T21:59:02.703 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.703 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.703 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.703 DEBUG:boto:Params: {} 2024-04-09T21:59:02.703 DEBUG:boto:Token: None 2024-04-09T21:59:02.703 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-13/?versions 2024-04-09T21:59:02.703 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vA1+zq0fuXhk3G0VaLi5q+xiSdc= 2024-04-09T21:59:02.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vA1+zq0fuXhk3G0VaLi5q+xiSdc=', 'Content-Length': '0'} 2024-04-09T21:59:02.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004beca34c09f86c71-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.706 DEBUG:boto:b'xryqqk-13obj-a1999falseobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.706 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:02.706 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:02.706 DEBUG:boto:path=/xryqqk-13/obj-a1?acl 2024-04-09T21:59:02.706 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?acl 2024-04-09T21:59:02.706 DEBUG:boto:Method: PUT 2024-04-09T21:59:02.706 DEBUG:boto:Path: /xryqqk-13/obj-a1?acl 2024-04-09T21:59:02.707 DEBUG:boto:Data: 2024-04-09T21:59:02.707 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T21:59:02.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.707 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.707 DEBUG:boto:Params: {} 2024-04-09T21:59:02.707 DEBUG:boto:Token: None 2024-04-09T21:59:02.707 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:59:02 GMT x-amz-acl:public-read /xryqqk-13/obj-a1?acl 2024-04-09T21:59:02.707 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:srnus/EQ8XhFYA6W+HXau+A5jwU= 2024-04-09T21:59:02.707 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:srnus/EQ8XhFYA6W+HXau+A5jwU=', 'Content-Length': '0'} 2024-04-09T21:59:02.713 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000304274c574f78663-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.713 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:02.713 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:02.713 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.713 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:59:02.713 DEBUG:boto:Data: 2024-04-09T21:59:02.713 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.714 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.714 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.714 DEBUG:boto:Params: {} 2024-04-09T21:59:02.714 DEBUG:boto:Token: None 2024-04-09T21:59:02.714 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-13/ 2024-04-09T21:59:02.714 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iB46DU8Bsohh2lg9BYjKS0ej2BM= 2024-04-09T21:59:02.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iB46DU8Bsohh2lg9BYjKS0ej2BM=', 'Content-Length': '0'} 2024-04-09T21:59:02.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e03c8fa1cf1aef1-006615ba26-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.717 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:02.717 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:02.717 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999&prefix=obj-a2 2024-04-09T21:59:02.717 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999&prefix=obj-a2 2024-04-09T21:59:02.717 DEBUG:boto:Method: GET 2024-04-09T21:59:02.717 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999&prefix=obj-a2 2024-04-09T21:59:02.717 DEBUG:boto:Data: 2024-04-09T21:59:02.717 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.717 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.717 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.718 DEBUG:boto:Params: {} 2024-04-09T21:59:02.718 DEBUG:boto:Token: None 2024-04-09T21:59:02.718 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-13/?versions 2024-04-09T21:59:02.718 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vA1+zq0fuXhk3G0VaLi5q+xiSdc= 2024-04-09T21:59:02.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vA1+zq0fuXhk3G0VaLi5q+xiSdc=', 'Content-Length': '0'} 2024-04-09T21:59:02.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026cf5321cf78d749-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.721 DEBUG:boto:b'xryqqk-13obj-a2999falseobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.721 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:02.721 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:02.721 DEBUG:boto:path=/xryqqk-13/obj-a2?acl 2024-04-09T21:59:02.721 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?acl 2024-04-09T21:59:02.721 DEBUG:boto:Method: PUT 2024-04-09T21:59:02.721 DEBUG:boto:Path: /xryqqk-13/obj-a2?acl 2024-04-09T21:59:02.721 DEBUG:boto:Data: 2024-04-09T21:59:02.721 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T21:59:02.722 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.722 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.722 DEBUG:boto:Params: {} 2024-04-09T21:59:02.722 DEBUG:boto:Token: None 2024-04-09T21:59:02.722 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:59:02 GMT x-amz-acl:public-read /xryqqk-13/obj-a2?acl 2024-04-09T21:59:02.722 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ov9fWHY6fVGyQf4GgdMFqulPohk= 2024-04-09T21:59:02.722 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ov9fWHY6fVGyQf4GgdMFqulPohk=', 'Content-Length': '0'} 2024-04-09T21:59:02.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078e1b0da6ebeb2e5-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.728 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.728 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.728 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.728 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:59:02.728 DEBUG:boto:Data: 2024-04-09T21:59:02.728 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.728 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.728 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.728 DEBUG:boto:Params: {} 2024-04-09T21:59:02.729 DEBUG:boto:Token: None 2024-04-09T21:59:02.729 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.729 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU= 2024-04-09T21:59:02.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU=', 'Content-Length': '0'} 2024-04-09T21:59:02.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001707c7f9a394d3e4-006615ba26-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.731 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.731 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.731 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999&prefix=obj-a1 2024-04-09T21:59:02.732 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999&prefix=obj-a1 2024-04-09T21:59:02.732 DEBUG:boto:Method: GET 2024-04-09T21:59:02.732 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999&prefix=obj-a1 2024-04-09T21:59:02.732 DEBUG:boto:Data: 2024-04-09T21:59:02.732 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.732 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.732 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.732 DEBUG:boto:Params: {} 2024-04-09T21:59:02.732 DEBUG:boto:Token: None 2024-04-09T21:59:02.732 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/?versions 2024-04-09T21:59:02.733 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20= 2024-04-09T21:59:02.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20=', 'Content-Length': '0'} 2024-04-09T21:59:02.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c4a2d5762a5a2ee-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.778 DEBUG:boto:b'xryqqk-14obj-a1999falseobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.779 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.779 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:02.779 DEBUG:boto:path=/xryqqk-14/obj-a1?acl 2024-04-09T21:59:02.779 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?acl 2024-04-09T21:59:02.779 DEBUG:boto:Method: PUT 2024-04-09T21:59:02.779 DEBUG:boto:Path: /xryqqk-14/obj-a1?acl 2024-04-09T21:59:02.779 DEBUG:boto:Data: 2024-04-09T21:59:02.779 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T21:59:02.780 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:02.780 DEBUG:boto:Port: 8000 2024-04-09T21:59:02.780 DEBUG:boto:Params: {} 2024-04-09T21:59:02.780 DEBUG:boto:Token: None 2024-04-09T21:59:02.780 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:59:02 GMT x-amz-acl:public-read /xryqqk-14/obj-a1?acl 2024-04-09T21:59:02.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LJSe3dnVomIUCw7SLmQBhX4U+Gc= 2024-04-09T21:59:02.780 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LJSe3dnVomIUCw7SLmQBhX4U+Gc=', 'Content-Length': '0'} 2024-04-09T21:59:02.786 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aebe0e80445e716d-006615ba26-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.786 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.786 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.787 DEBUG:boto:Method: HEAD 2024-04-09T21:59:02.787 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:59:02.787 DEBUG:boto:Data: 2024-04-09T21:59:02.787 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.787 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.787 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.787 DEBUG:boto:Params: {} 2024-04-09T21:59:02.787 DEBUG:boto:Token: None 2024-04-09T21:59:02.787 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/ 2024-04-09T21:59:02.787 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU= 2024-04-09T21:59:02.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y6Nfq37wXKr80O+DNvYaR7DO6JU=', 'Content-Length': '0'} 2024-04-09T21:59:02.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d091ce6ee6de563-006615ba26-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.790 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:02.790 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:02.790 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999&prefix=obj-a2 2024-04-09T21:59:02.790 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999&prefix=obj-a2 2024-04-09T21:59:02.790 DEBUG:boto:Method: GET 2024-04-09T21:59:02.790 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999&prefix=obj-a2 2024-04-09T21:59:02.790 DEBUG:boto:Data: 2024-04-09T21:59:02.790 DEBUG:boto:Headers: {} 2024-04-09T21:59:02.790 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.791 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.791 DEBUG:boto:Params: {} 2024-04-09T21:59:02.791 DEBUG:boto:Token: None 2024-04-09T21:59:02.791 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:02 GMT /xryqqk-14/?versions 2024-04-09T21:59:02.791 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20= 2024-04-09T21:59:02.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JuUCFiiHcNzFGkw0/dKiEwuLV20=', 'Content-Length': '0'} 2024-04-09T21:59:02.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035aff8a5111b090f-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.793 DEBUG:boto:b'xryqqk-14obj-a2999falseobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:02.794 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.794 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:02.794 DEBUG:boto:path=/xryqqk-14/obj-a2?acl 2024-04-09T21:59:02.794 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?acl 2024-04-09T21:59:02.794 DEBUG:boto:Method: PUT 2024-04-09T21:59:02.794 DEBUG:boto:Path: /xryqqk-14/obj-a2?acl 2024-04-09T21:59:02.794 DEBUG:boto:Data: 2024-04-09T21:59:02.794 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T21:59:02.794 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:02.794 DEBUG:boto:Port: 8001 2024-04-09T21:59:02.794 DEBUG:boto:Params: {} 2024-04-09T21:59:02.795 DEBUG:boto:Token: None 2024-04-09T21:59:02.795 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:59:02 GMT x-amz-acl:public-read /xryqqk-14/obj-a2?acl 2024-04-09T21:59:02.795 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XQOfTaTmSb4cYGGiUuIfHEImB8c= 2024-04-09T21:59:02.795 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XQOfTaTmSb4cYGGiUuIfHEImB8c=', 'Content-Length': '0'} 2024-04-09T21:59:02.802 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000195931d3f8483eee-006615ba26-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:02.802 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-13 2024-04-09T21:59:02.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:02.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:02.803 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:02.860 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:02.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:02.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:02.895+0000 7f58fd2b9780 1 waiting for incremental sync to catch up: 2024-04-09T21:59:02.897 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000010.22.6, ] 2024-04-09T21:59:02.897 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000011.25.6, ] 2024-04-09T21:59:07.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:07.897+0000 7f58fd2b9780 1 waiting for incremental sync to catch up: 2024-04-09T21:59:07.898 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000010.22.6, ] 2024-04-09T21:59:07.899 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000011.25.6, ] 2024-04-09T21:59:12.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:12.898+0000 7f58fd2b9780 1 waiting for incremental sync to catch up: 2024-04-09T21:59:12.900 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000010.22.6, ] 2024-04-09T21:59:12.900 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000011.25.6, ] 2024-04-09T21:59:17.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:17.900+0000 7f58fd2b9780 1 waiting for incremental sync to catch up: 2024-04-09T21:59:17.901 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000010.22.6, ] 2024-04-09T21:59:17.901 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000011.25.6, ] 2024-04-09T21:59:22.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:22.901+0000 7f58fd2b9780 1 bucket sync caught up with source: 2024-04-09T21:59:22.903 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000011.25.6, ] 2024-04-09T21:59:22.903 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000002.16.11, 00000000011.25.6, ] 2024-04-09T21:59:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:22.901+0000 7f58fd2b9780 0 bucket checkpoint complete 2024-04-09T21:59:22.919 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-13 2024-04-09T21:59:22.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:22.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:22.920 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:22.982 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:22.982 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:23.020+0000 7f0911088780 1 bucket sync caught up with source: 2024-04-09T21:59:23.021 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , 00000000011.41.6, 00000000002.11.11, ] 2024-04-09T21:59:23.021 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , 00000000011.41.6, 00000000002.11.11, ] 2024-04-09T21:59:23.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:23.020+0000 7f0911088780 0 bucket checkpoint complete 2024-04-09T21:59:23.034 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-13 zones={a2, a1} 2024-04-09T21:59:23.034 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.034 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.034 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.035 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:59:23.035 DEBUG:boto:Data: 2024-04-09T21:59:23.035 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.035 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.035 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.035 DEBUG:boto:Params: {} 2024-04-09T21:59:23.035 DEBUG:boto:Token: None 2024-04-09T21:59:23.035 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.036 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:q/NIH6hqDpQqBNqAbMwzaoTPxco= 2024-04-09T21:59:23.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:q/NIH6hqDpQqBNqAbMwzaoTPxco=', 'Content-Length': '0'} 2024-04-09T21:59:23.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000827a8e58ee46c075-006615ba3b-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.039 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.039 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.039 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.039 DEBUG:boto:Path: /xryqqk-13/ 2024-04-09T21:59:23.039 DEBUG:boto:Data: 2024-04-09T21:59:23.039 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.039 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.039 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.039 DEBUG:boto:Params: {} 2024-04-09T21:59:23.040 DEBUG:boto:Token: None 2024-04-09T21:59:23.040 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.040 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:q/NIH6hqDpQqBNqAbMwzaoTPxco= 2024-04-09T21:59:23.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:q/NIH6hqDpQqBNqAbMwzaoTPxco=', 'Content-Length': '0'} 2024-04-09T21:59:23.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000501757411866d489-006615ba3b-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.042 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:59:23.042 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.042 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.042 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.043 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.043 DEBUG:boto:Method: GET 2024-04-09T21:59:23.043 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.043 DEBUG:boto:Data: 2024-04-09T21:59:23.043 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.043 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.043 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.043 DEBUG:boto:Params: {} 2024-04-09T21:59:23.043 DEBUG:boto:Token: None 2024-04-09T21:59:23.043 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/?versions 2024-04-09T21:59:23.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo= 2024-04-09T21:59:23.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo=', 'Content-Length': '0'} 2024-04-09T21:59:23.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d108e29df8cce5-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.046 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.047 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:23.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:23.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:23.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:59:23.047 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.047 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.047 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.048 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.048 DEBUG:boto:Method: GET 2024-04-09T21:59:23.048 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.048 DEBUG:boto:Data: 2024-04-09T21:59:23.048 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.048 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.048 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.048 DEBUG:boto:Params: {} 2024-04-09T21:59:23.048 DEBUG:boto:Token: None 2024-04-09T21:59:23.049 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/?versions 2024-04-09T21:59:23.049 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo= 2024-04-09T21:59:23.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo=', 'Content-Length': '0'} 2024-04-09T21:59:23.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098220119e067071b-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.051 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.052 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:23.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:23.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:23.052 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.052 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.052 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.052 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.052 DEBUG:boto:Method: GET 2024-04-09T21:59:23.052 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.052 DEBUG:boto:Data: 2024-04-09T21:59:23.053 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.053 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.053 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.053 DEBUG:boto:Params: {} 2024-04-09T21:59:23.053 DEBUG:boto:Token: None 2024-04-09T21:59:23.053 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/?versions 2024-04-09T21:59:23.053 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo= 2024-04-09T21:59:23.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo=', 'Content-Length': '0'} 2024-04-09T21:59:23.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cac3f248e1a9f2b-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.056 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.056 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.056 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.056 DEBUG:boto:path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.056 DEBUG:boto:auth_path=/xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.056 DEBUG:boto:Method: GET 2024-04-09T21:59:23.056 DEBUG:boto:Path: /xryqqk-13/?versions&max-keys=999 2024-04-09T21:59:23.057 DEBUG:boto:Data: 2024-04-09T21:59:23.057 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.057 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.057 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.057 DEBUG:boto:Params: {} 2024-04-09T21:59:23.057 DEBUG:boto:Token: None 2024-04-09T21:59:23.057 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/?versions 2024-04-09T21:59:23.057 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo= 2024-04-09T21:59:23.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dfbLZ1Z+K/Gt0tR8TcqlAh8KAFo=', 'Content-Length': '0'} 2024-04-09T21:59:23.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc8c56b850f4496c-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.060 DEBUG:boto:b'xryqqk-13999falsedummyUgVCTjwaf-RNeS4vdANijkTMDGSICUTtrue2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1As-ZAgS59ojuh3nOMPpE01Os9z1-jO7true2024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzDtrue2024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:23.061 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.061 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.061 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.061 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.061 DEBUG:boto:Method: GET 2024-04-09T21:59:23.061 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.061 DEBUG:boto:Data: 2024-04-09T21:59:23.061 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.061 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.061 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.062 DEBUG:boto:Params: {} 2024-04-09T21:59:23.062 DEBUG:boto:Token: None 2024-04-09T21:59:23.062 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.062 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g= 2024-04-09T21:59:23.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g=', 'Content-Length': '0'} 2024-04-09T21:59:23.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d48c657dc8883aee-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.064 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.064 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.064 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.064 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.064 DEBUG:boto:Method: GET 2024-04-09T21:59:23.064 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.064 DEBUG:boto:Data: 2024-04-09T21:59:23.064 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.065 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.065 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.065 DEBUG:boto:Params: {} 2024-04-09T21:59:23.065 DEBUG:boto:Token: None 2024-04-09T21:59:23.065 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.065 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g= 2024-04-09T21:59:23.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g=', 'Content-Length': '0'} 2024-04-09T21:59:23.067 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d624025853a7d99-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.067 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.067 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.067 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.067 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.067 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.067 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.067 DEBUG:boto:Data: 2024-04-09T21:59:23.068 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.068 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.068 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.068 DEBUG:boto:Params: {} 2024-04-09T21:59:23.068 DEBUG:boto:Token: None 2024-04-09T21:59:23.068 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.068 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sAr8kVZQfQv7uvVgNCApW95EgX4= 2024-04-09T21:59:23.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sAr8kVZQfQv7uvVgNCApW95EgX4=', 'Content-Length': '0'} 2024-04-09T21:59:23.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000912ededbf399600e-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.070 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.070 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.070 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.070 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.070 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.071 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.071 DEBUG:boto:Data: 2024-04-09T21:59:23.071 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.071 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.071 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.071 DEBUG:boto:Params: {} 2024-04-09T21:59:23.071 DEBUG:boto:Token: None 2024-04-09T21:59:23.071 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.071 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sAr8kVZQfQv7uvVgNCApW95EgX4= 2024-04-09T21:59:23.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sAr8kVZQfQv7uvVgNCApW95EgX4=', 'Content-Length': '0'} 2024-04-09T21:59:23.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e60af94a1cc0ab7d-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:23.073 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.073 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.073 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.074 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.074 DEBUG:boto:Method: GET 2024-04-09T21:59:23.074 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.074 DEBUG:boto:Data: 2024-04-09T21:59:23.074 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.074 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.074 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.074 DEBUG:boto:Params: {} 2024-04-09T21:59:23.074 DEBUG:boto:Token: None 2024-04-09T21:59:23.074 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.075 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g= 2024-04-09T21:59:23.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g=', 'Content-Length': '0'} 2024-04-09T21:59:23.076 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004099f561f5131b8a-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.076 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.076 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.076 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.077 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.077 DEBUG:boto:Method: GET 2024-04-09T21:59:23.077 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.077 DEBUG:boto:Data: 2024-04-09T21:59:23.077 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.077 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.077 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.077 DEBUG:boto:Params: {} 2024-04-09T21:59:23.077 DEBUG:boto:Token: None 2024-04-09T21:59:23.077 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.078 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g= 2024-04-09T21:59:23.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g=', 'Content-Length': '0'} 2024-04-09T21:59:23.079 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005bf5f80e730a849d-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.079 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.079 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.080 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.080 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.080 DEBUG:boto:Method: GET 2024-04-09T21:59:23.080 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.080 DEBUG:boto:Data: 2024-04-09T21:59:23.080 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.080 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.080 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.080 DEBUG:boto:Params: {} 2024-04-09T21:59:23.080 DEBUG:boto:Token: None 2024-04-09T21:59:23.081 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.081 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093e158d28f99c793-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.083 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.083 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.084 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.084 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.084 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.084 DEBUG:boto:Method: GET 2024-04-09T21:59:23.084 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.084 DEBUG:boto:Data: 2024-04-09T21:59:23.084 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.084 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.084 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.084 DEBUG:boto:Params: {} 2024-04-09T21:59:23.084 DEBUG:boto:Token: None 2024-04-09T21:59:23.084 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.085 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019ddcf8ffb550bce-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.130 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.131 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.131 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.131 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.131 DEBUG:boto:Path: /xryqqk-13/dummy 2024-04-09T21:59:23.131 DEBUG:boto:Data: 2024-04-09T21:59:23.131 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.131 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.131 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.132 DEBUG:boto:Params: {} 2024-04-09T21:59:23.132 DEBUG:boto:Token: None 2024-04-09T21:59:23.132 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy 2024-04-09T21:59:23.132 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zMjra0eQFLIEX4FVO69upPUEXow= 2024-04-09T21:59:23.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zMjra0eQFLIEX4FVO69upPUEXow=', 'Content-Length': '0'} 2024-04-09T21:59:23.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a65e0c45377992c8-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.134 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.134 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.134 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.134 DEBUG:boto:Path: /xryqqk-13/dummy 2024-04-09T21:59:23.135 DEBUG:boto:Data: 2024-04-09T21:59:23.135 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.135 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.135 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.135 DEBUG:boto:Params: {} 2024-04-09T21:59:23.135 DEBUG:boto:Token: None 2024-04-09T21:59:23.135 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy 2024-04-09T21:59:23.135 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zMjra0eQFLIEX4FVO69upPUEXow= 2024-04-09T21:59:23.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zMjra0eQFLIEX4FVO69upPUEXow=', 'Content-Length': '0'} 2024-04-09T21:59:23.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d576c4017ac6705-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:23.138 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.138 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.138 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.138 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.138 DEBUG:boto:Method: GET 2024-04-09T21:59:23.138 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.138 DEBUG:boto:Data: 2024-04-09T21:59:23.138 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.138 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.138 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.138 DEBUG:boto:Params: {} 2024-04-09T21:59:23.139 DEBUG:boto:Token: None 2024-04-09T21:59:23.139 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.139 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g= 2024-04-09T21:59:23.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g=', 'Content-Length': '0'} 2024-04-09T21:59:23.140 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039579da3e8ff3b85-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.140 DEBUG:boto:path=/xryqqk-13/dummy 2024-04-09T21:59:23.141 DEBUG:boto:auth_path=/xryqqk-13/dummy 2024-04-09T21:59:23.141 DEBUG:boto:path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.141 DEBUG:boto:auth_path=/xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.141 DEBUG:boto:Method: GET 2024-04-09T21:59:23.141 DEBUG:boto:Path: /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.141 DEBUG:boto:Data: 2024-04-09T21:59:23.141 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.141 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.141 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.141 DEBUG:boto:Params: {} 2024-04-09T21:59:23.141 DEBUG:boto:Token: None 2024-04-09T21:59:23.141 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/dummy?versionId=UgVCTjwaf-RNeS4vdANijkTMDGSICUT 2024-04-09T21:59:23.142 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g= 2024-04-09T21:59:23.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wOE7KM7HwR7vWT+2szuw3CNWS/g=', 'Content-Length': '0'} 2024-04-09T21:59:23.143 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:57:45 GMT'), ('x-amz-version-id', 'UgVCTjwaf-RNeS4vdANijkTMDGSICUT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007749570d02c0935b-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.143 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.143 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.144 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.144 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.144 DEBUG:boto:Method: GET 2024-04-09T21:59:23.144 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.144 DEBUG:boto:Data: 2024-04-09T21:59:23.144 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.144 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.144 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.144 DEBUG:boto:Params: {} 2024-04-09T21:59:23.144 DEBUG:boto:Token: None 2024-04-09T21:59:23.144 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.145 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0aa6cc166ed88e6-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.194 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.195 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.195 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.195 DEBUG:boto:path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.195 DEBUG:boto:auth_path=/xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.195 DEBUG:boto:Method: GET 2024-04-09T21:59:23.195 DEBUG:boto:Path: /xryqqk-13/?prefix=dummy 2024-04-09T21:59:23.195 DEBUG:boto:Data: 2024-04-09T21:59:23.195 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.195 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.195 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.195 DEBUG:boto:Params: {} 2024-04-09T21:59:23.196 DEBUG:boto:Token: None 2024-04-09T21:59:23.196 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.196 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b626a20a6c19c38-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.198 DEBUG:boto:b'xryqqk-13dummy1000falsedummy2024-04-09T21:57:45.467Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:23.199 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.199 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.199 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.199 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.199 DEBUG:boto:Method: GET 2024-04-09T21:59:23.199 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.199 DEBUG:boto:Data: 2024-04-09T21:59:23.199 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.199 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.200 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.200 DEBUG:boto:Params: {} 2024-04-09T21:59:23.200 DEBUG:boto:Token: None 2024-04-09T21:59:23.200 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.200 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU= 2024-04-09T21:59:23.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU=', 'Content-Length': '0'} 2024-04-09T21:59:23.201 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000081afa6c1df1d88ff-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.202 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.202 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.202 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.202 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.202 DEBUG:boto:Method: GET 2024-04-09T21:59:23.203 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.203 DEBUG:boto:Data: 2024-04-09T21:59:23.203 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.203 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.203 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.203 DEBUG:boto:Params: {} 2024-04-09T21:59:23.203 DEBUG:boto:Token: None 2024-04-09T21:59:23.203 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.203 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU= 2024-04-09T21:59:23.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU=', 'Content-Length': '0'} 2024-04-09T21:59:23.205 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed938f8eaf213084-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.205 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.205 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.205 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.205 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.205 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.205 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.206 DEBUG:boto:Data: 2024-04-09T21:59:23.206 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.206 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.206 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.206 DEBUG:boto:Params: {} 2024-04-09T21:59:23.206 DEBUG:boto:Token: None 2024-04-09T21:59:23.209 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.210 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iRmztBh53X4Kz7JapMcJT0UQ26U= 2024-04-09T21:59:23.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iRmztBh53X4Kz7JapMcJT0UQ26U=', 'Content-Length': '0'} 2024-04-09T21:59:23.211 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006e1104cf066309fa-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.211 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.212 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.212 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.212 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.212 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.212 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.212 DEBUG:boto:Data: 2024-04-09T21:59:23.212 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.212 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.212 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.212 DEBUG:boto:Params: {} 2024-04-09T21:59:23.212 DEBUG:boto:Token: None 2024-04-09T21:59:23.213 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.213 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iRmztBh53X4Kz7JapMcJT0UQ26U= 2024-04-09T21:59:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iRmztBh53X4Kz7JapMcJT0UQ26U=', 'Content-Length': '0'} 2024-04-09T21:59:23.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f25f09a6316c86ba-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:23.215 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.215 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.215 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.215 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.215 DEBUG:boto:Method: GET 2024-04-09T21:59:23.215 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.215 DEBUG:boto:Data: 2024-04-09T21:59:23.215 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.216 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.216 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.216 DEBUG:boto:Params: {} 2024-04-09T21:59:23.216 DEBUG:boto:Token: None 2024-04-09T21:59:23.216 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.216 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU= 2024-04-09T21:59:23.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU=', 'Content-Length': '0'} 2024-04-09T21:59:23.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f77f2a869d88ee61-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.262 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.262 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.263 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.263 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.263 DEBUG:boto:Method: GET 2024-04-09T21:59:23.263 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.263 DEBUG:boto:Data: 2024-04-09T21:59:23.263 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.263 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.263 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.263 DEBUG:boto:Params: {} 2024-04-09T21:59:23.263 DEBUG:boto:Token: None 2024-04-09T21:59:23.264 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.264 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU= 2024-04-09T21:59:23.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU=', 'Content-Length': '0'} 2024-04-09T21:59:23.265 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3cca8d2138ef7e8-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.266 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.266 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.266 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.266 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.266 DEBUG:boto:Method: GET 2024-04-09T21:59:23.266 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.266 DEBUG:boto:Data: 2024-04-09T21:59:23.266 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.266 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.267 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.267 DEBUG:boto:Params: {} 2024-04-09T21:59:23.267 DEBUG:boto:Token: None 2024-04-09T21:59:23.267 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.267 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c484ec04d34bacf-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.269 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.270 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.270 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.270 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.270 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.270 DEBUG:boto:Method: GET 2024-04-09T21:59:23.270 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.270 DEBUG:boto:Data: 2024-04-09T21:59:23.271 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.271 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.271 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.271 DEBUG:boto:Params: {} 2024-04-09T21:59:23.271 DEBUG:boto:Token: None 2024-04-09T21:59:23.271 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.271 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa7fdd01d737f14-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.274 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.274 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.274 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.274 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.274 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:59:23.274 DEBUG:boto:Data: 2024-04-09T21:59:23.275 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.275 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.275 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.275 DEBUG:boto:Params: {} 2024-04-09T21:59:23.275 DEBUG:boto:Token: None 2024-04-09T21:59:23.275 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1 2024-04-09T21:59:23.275 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aQ7Cl7JqL/EnEiY7l9lVqjos2O8= 2024-04-09T21:59:23.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aQ7Cl7JqL/EnEiY7l9lVqjos2O8=', 'Content-Length': '0'} 2024-04-09T21:59:23.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001731d36e339902e2-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.277 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.277 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.277 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.277 DEBUG:boto:Path: /xryqqk-13/obj-a1 2024-04-09T21:59:23.278 DEBUG:boto:Data: 2024-04-09T21:59:23.278 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.278 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.278 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.278 DEBUG:boto:Params: {} 2024-04-09T21:59:23.278 DEBUG:boto:Token: None 2024-04-09T21:59:23.278 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1 2024-04-09T21:59:23.278 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aQ7Cl7JqL/EnEiY7l9lVqjos2O8= 2024-04-09T21:59:23.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aQ7Cl7JqL/EnEiY7l9lVqjos2O8=', 'Content-Length': '0'} 2024-04-09T21:59:23.280 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb1c184e5692f57d-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:23.281 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.281 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.281 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.281 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.281 DEBUG:boto:Method: GET 2024-04-09T21:59:23.281 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.281 DEBUG:boto:Data: 2024-04-09T21:59:23.281 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.281 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.281 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.281 DEBUG:boto:Params: {} 2024-04-09T21:59:23.282 DEBUG:boto:Token: None 2024-04-09T21:59:23.282 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.282 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU= 2024-04-09T21:59:23.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU=', 'Content-Length': '0'} 2024-04-09T21:59:23.283 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003256fb648fc9cf7b-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.326 DEBUG:boto:path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.327 DEBUG:boto:auth_path=/xryqqk-13/obj-a1 2024-04-09T21:59:23.327 DEBUG:boto:path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.327 DEBUG:boto:auth_path=/xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.327 DEBUG:boto:Method: GET 2024-04-09T21:59:23.327 DEBUG:boto:Path: /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.327 DEBUG:boto:Data: 2024-04-09T21:59:23.327 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.327 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.327 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.327 DEBUG:boto:Params: {} 2024-04-09T21:59:23.328 DEBUG:boto:Token: None 2024-04-09T21:59:23.328 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a1?versionId=As-ZAgS59ojuh3nOMPpE01Os9z1-jO7 2024-04-09T21:59:23.328 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU= 2024-04-09T21:59:23.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GGUt2/vU2NPkIDcwPBNm7CRlGU=', 'Content-Length': '0'} 2024-04-09T21:59:23.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'As-ZAgS59ojuh3nOMPpE01Os9z1-jO7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9a3875d978d6941-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.330 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.330 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.330 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.330 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.330 DEBUG:boto:Method: GET 2024-04-09T21:59:23.330 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.330 DEBUG:boto:Data: 2024-04-09T21:59:23.330 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.331 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.331 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.331 DEBUG:boto:Params: {} 2024-04-09T21:59:23.331 DEBUG:boto:Token: None 2024-04-09T21:59:23.331 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.331 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a95ea456638ac1a-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.378 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.379 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.379 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.379 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.379 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.379 DEBUG:boto:Method: GET 2024-04-09T21:59:23.379 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a1 2024-04-09T21:59:23.379 DEBUG:boto:Data: 2024-04-09T21:59:23.380 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.380 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.380 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.380 DEBUG:boto:Params: {} 2024-04-09T21:59:23.380 DEBUG:boto:Token: None 2024-04-09T21:59:23.380 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.380 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f93f0b0d4f09564-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.383 DEBUG:boto:b'xryqqk-13obj-a11000falseobj-a12024-04-09T21:58:20.734Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:23.384 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.384 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.384 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.384 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.384 DEBUG:boto:Method: GET 2024-04-09T21:59:23.384 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.384 DEBUG:boto:Data: 2024-04-09T21:59:23.384 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.384 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.384 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.384 DEBUG:boto:Params: {} 2024-04-09T21:59:23.385 DEBUG:boto:Token: None 2024-04-09T21:59:23.385 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.385 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U= 2024-04-09T21:59:23.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U=', 'Content-Length': '0'} 2024-04-09T21:59:23.386 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3bd342b46672339-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.386 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.387 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.387 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.387 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.387 DEBUG:boto:Method: GET 2024-04-09T21:59:23.387 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.387 DEBUG:boto:Data: 2024-04-09T21:59:23.387 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.387 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.387 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.387 DEBUG:boto:Params: {} 2024-04-09T21:59:23.387 DEBUG:boto:Token: None 2024-04-09T21:59:23.388 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.388 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U= 2024-04-09T21:59:23.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U=', 'Content-Length': '0'} 2024-04-09T21:59:23.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004f8fc70e926d4de5-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.430 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.431 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.431 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.431 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.431 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.431 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.431 DEBUG:boto:Data: 2024-04-09T21:59:23.431 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.431 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.431 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.431 DEBUG:boto:Params: {} 2024-04-09T21:59:23.431 DEBUG:boto:Token: None 2024-04-09T21:59:23.432 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.432 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xACUBxLLuWcwFK4gXsNFYyisW/0= 2024-04-09T21:59:23.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xACUBxLLuWcwFK4gXsNFYyisW/0=', 'Content-Length': '0'} 2024-04-09T21:59:23.433 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e83fcd6bc892a0a-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.434 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.434 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.434 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.434 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.434 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.434 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.434 DEBUG:boto:Data: 2024-04-09T21:59:23.434 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.435 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.435 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.435 DEBUG:boto:Params: {} 2024-04-09T21:59:23.435 DEBUG:boto:Token: None 2024-04-09T21:59:23.435 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.435 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xACUBxLLuWcwFK4gXsNFYyisW/0= 2024-04-09T21:59:23.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xACUBxLLuWcwFK4gXsNFYyisW/0=', 'Content-Length': '0'} 2024-04-09T21:59:23.436 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd8e599b6303ed32-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:23.437 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.437 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.437 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.437 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.437 DEBUG:boto:Method: GET 2024-04-09T21:59:23.437 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.437 DEBUG:boto:Data: 2024-04-09T21:59:23.437 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.437 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.438 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.438 DEBUG:boto:Params: {} 2024-04-09T21:59:23.438 DEBUG:boto:Token: None 2024-04-09T21:59:23.438 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.438 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U= 2024-04-09T21:59:23.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U=', 'Content-Length': '0'} 2024-04-09T21:59:23.440 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd0da54fa18c883c-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.440 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.440 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.440 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.440 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.440 DEBUG:boto:Method: GET 2024-04-09T21:59:23.440 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.440 DEBUG:boto:Data: 2024-04-09T21:59:23.441 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.441 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.441 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.441 DEBUG:boto:Params: {} 2024-04-09T21:59:23.441 DEBUG:boto:Token: None 2024-04-09T21:59:23.441 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.441 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U= 2024-04-09T21:59:23.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U=', 'Content-Length': '0'} 2024-04-09T21:59:23.443 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051dbfb413b2a7fff-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.486 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.487 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.487 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.487 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.487 DEBUG:boto:Method: GET 2024-04-09T21:59:23.487 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.487 DEBUG:boto:Data: 2024-04-09T21:59:23.487 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.487 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.487 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.487 DEBUG:boto:Params: {} 2024-04-09T21:59:23.488 DEBUG:boto:Token: None 2024-04-09T21:59:23.488 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.488 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191d240cabdf6e2b-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.490 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.491 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.491 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.491 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.491 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.491 DEBUG:boto:Method: GET 2024-04-09T21:59:23.491 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.491 DEBUG:boto:Data: 2024-04-09T21:59:23.491 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.491 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.491 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.491 DEBUG:boto:Params: {} 2024-04-09T21:59:23.492 DEBUG:boto:Token: None 2024-04-09T21:59:23.492 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.492 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1ff334feb0a3d6d-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.494 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.494 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.495 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.495 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.495 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:59:23.495 DEBUG:boto:Data: 2024-04-09T21:59:23.495 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.495 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.495 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.495 DEBUG:boto:Params: {} 2024-04-09T21:59:23.495 DEBUG:boto:Token: None 2024-04-09T21:59:23.495 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2 2024-04-09T21:59:23.496 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cqfVTPtkLHH1pnqv7FO8xCANv1s= 2024-04-09T21:59:23.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cqfVTPtkLHH1pnqv7FO8xCANv1s=', 'Content-Length': '0'} 2024-04-09T21:59:23.498 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066422b1feb338862-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.498 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.498 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.498 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.498 DEBUG:boto:Path: /xryqqk-13/obj-a2 2024-04-09T21:59:23.498 DEBUG:boto:Data: 2024-04-09T21:59:23.498 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.498 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.498 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.499 DEBUG:boto:Params: {} 2024-04-09T21:59:23.499 DEBUG:boto:Token: None 2024-04-09T21:59:23.499 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2 2024-04-09T21:59:23.499 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cqfVTPtkLHH1pnqv7FO8xCANv1s= 2024-04-09T21:59:23.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cqfVTPtkLHH1pnqv7FO8xCANv1s=', 'Content-Length': '0'} 2024-04-09T21:59:23.501 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004400a75efdd74acf-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:23.501 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.501 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.501 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.501 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.502 DEBUG:boto:Method: GET 2024-04-09T21:59:23.502 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.502 DEBUG:boto:Data: 2024-04-09T21:59:23.502 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.502 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.502 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.502 DEBUG:boto:Params: {} 2024-04-09T21:59:23.502 DEBUG:boto:Token: None 2024-04-09T21:59:23.502 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.503 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U= 2024-04-09T21:59:23.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U=', 'Content-Length': '0'} 2024-04-09T21:59:23.504 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2f6b6dd7ae7ffe1-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.504 DEBUG:boto:path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.504 DEBUG:boto:auth_path=/xryqqk-13/obj-a2 2024-04-09T21:59:23.505 DEBUG:boto:path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.505 DEBUG:boto:auth_path=/xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.505 DEBUG:boto:Method: GET 2024-04-09T21:59:23.505 DEBUG:boto:Path: /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.505 DEBUG:boto:Data: 2024-04-09T21:59:23.505 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.505 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.505 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.505 DEBUG:boto:Params: {} 2024-04-09T21:59:23.505 DEBUG:boto:Token: None 2024-04-09T21:59:23.506 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/obj-a2?versionId=ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD 2024-04-09T21:59:23.506 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U= 2024-04-09T21:59:23.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/+AOQyHlbEptnkgZXYL8BhQEV9U=', 'Content-Length': '0'} 2024-04-09T21:59:23.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'ux-tMUgrTTiv2H8IwKt9z9M9pWGFEzD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-13:a5723776-f142-428c-a6df-d7bf949257da.4372.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072eda27d52d8eee8-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.551 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.551 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.551 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.551 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.551 DEBUG:boto:Method: GET 2024-04-09T21:59:23.551 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.551 DEBUG:boto:Data: 2024-04-09T21:59:23.551 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.551 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.552 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.552 DEBUG:boto:Params: {} 2024-04-09T21:59:23.552 DEBUG:boto:Token: None 2024-04-09T21:59:23.552 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.552 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047e9f7fd0389767e-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.555 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.555 DEBUG:boto:path=/xryqqk-13/ 2024-04-09T21:59:23.555 DEBUG:boto:auth_path=/xryqqk-13/ 2024-04-09T21:59:23.556 DEBUG:boto:path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.556 DEBUG:boto:auth_path=/xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.556 DEBUG:boto:Method: GET 2024-04-09T21:59:23.556 DEBUG:boto:Path: /xryqqk-13/?prefix=obj-a2 2024-04-09T21:59:23.556 DEBUG:boto:Data: 2024-04-09T21:59:23.556 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.556 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.556 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.556 DEBUG:boto:Params: {} 2024-04-09T21:59:23.556 DEBUG:boto:Token: None 2024-04-09T21:59:23.557 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-13/ 2024-04-09T21:59:23.557 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik= 2024-04-09T21:59:23.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UKW7We9mkAG2c0g39oip9nCMJik=', 'Content-Length': '0'} 2024-04-09T21:59:23.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091c72a1e8e222963-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.559 DEBUG:boto:b'xryqqk-13obj-a21000falseobj-a22024-04-09T21:58:20.861Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.560 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-13 zones={a2, a1} 2024-04-09T21:59:23.560 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-14 2024-04-09T21:59:23.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:23.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:23.560 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:23.618 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:23.618 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:23.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:23.654+0000 7f9d78120780 1 bucket sync caught up with source: 2024-04-09T21:59:23.655 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.9.11, , , 00000000002.5.11, 00000000011.26.6, ] 2024-04-09T21:59:23.655 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.9.11, , , 00000000002.5.11, 00000000011.26.6, ] 2024-04-09T21:59:23.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:23.654+0000 7f9d78120780 0 bucket checkpoint complete 2024-04-09T21:59:23.667 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-14 2024-04-09T21:59:23.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:23.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:23.668 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:23.724 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:23.724 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:23.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:23.758+0000 7fee25351780 1 bucket sync caught up with source: 2024-04-09T21:59:23.759 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.18.10, , , 00000000011.27.6, 00000000002.13.11, ] 2024-04-09T21:59:23.759 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.18.10, , , 00000000011.27.6, 00000000002.13.11, ] 2024-04-09T21:59:23.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:23.758+0000 7fee25351780 0 bucket checkpoint complete 2024-04-09T21:59:23.772 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-14 zones={a2, a1} 2024-04-09T21:59:23.772 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.772 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.772 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.772 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:59:23.772 DEBUG:boto:Data: 2024-04-09T21:59:23.772 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.773 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.773 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.773 DEBUG:boto:Params: {} 2024-04-09T21:59:23.773 DEBUG:boto:Token: None 2024-04-09T21:59:23.773 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.773 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:P9In7Rlfrfd85DVu1no849o5Kv8= 2024-04-09T21:59:23.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:P9In7Rlfrfd85DVu1no849o5Kv8=', 'Content-Length': '0'} 2024-04-09T21:59:23.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5c7aa0c17be02b4-006615ba3b-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.776 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.776 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.776 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.776 DEBUG:boto:Path: /xryqqk-14/ 2024-04-09T21:59:23.776 DEBUG:boto:Data: 2024-04-09T21:59:23.776 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.776 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.776 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.776 DEBUG:boto:Params: {} 2024-04-09T21:59:23.777 DEBUG:boto:Token: None 2024-04-09T21:59:23.777 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.777 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:P9In7Rlfrfd85DVu1no849o5Kv8= 2024-04-09T21:59:23.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:P9In7Rlfrfd85DVu1no849o5Kv8=', 'Content-Length': '0'} 2024-04-09T21:59:23.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007898de6d96041818-006615ba3b-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.779 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:59:23.779 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.779 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.779 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.780 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.780 DEBUG:boto:Method: GET 2024-04-09T21:59:23.780 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.780 DEBUG:boto:Data: 2024-04-09T21:59:23.780 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.780 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.780 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.780 DEBUG:boto:Params: {} 2024-04-09T21:59:23.780 DEBUG:boto:Token: None 2024-04-09T21:59:23.780 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/?versions 2024-04-09T21:59:23.781 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ= 2024-04-09T21:59:23.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ=', 'Content-Length': '0'} 2024-04-09T21:59:23.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbbf179f0e7026d-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.783 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.783 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:23.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:23.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:23.784 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:59:23.784 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.784 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.784 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.784 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.784 DEBUG:boto:Method: GET 2024-04-09T21:59:23.784 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.784 DEBUG:boto:Data: 2024-04-09T21:59:23.784 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.784 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.784 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.785 DEBUG:boto:Params: {} 2024-04-09T21:59:23.785 DEBUG:boto:Token: None 2024-04-09T21:59:23.785 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/?versions 2024-04-09T21:59:23.785 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ= 2024-04-09T21:59:23.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ=', 'Content-Length': '0'} 2024-04-09T21:59:23.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e911e31f0668452a-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.788 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.788 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:23.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-09T21:59:23.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-09T21:59:23.788 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.788 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.789 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.789 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.789 DEBUG:boto:Method: GET 2024-04-09T21:59:23.789 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.789 DEBUG:boto:Data: 2024-04-09T21:59:23.789 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.789 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.789 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.789 DEBUG:boto:Params: {} 2024-04-09T21:59:23.789 DEBUG:boto:Token: None 2024-04-09T21:59:23.789 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/?versions 2024-04-09T21:59:23.790 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ= 2024-04-09T21:59:23.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ=', 'Content-Length': '0'} 2024-04-09T21:59:23.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064943121e7fc0797-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.792 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.792 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.792 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.793 DEBUG:boto:path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.793 DEBUG:boto:auth_path=/xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.793 DEBUG:boto:Method: GET 2024-04-09T21:59:23.793 DEBUG:boto:Path: /xryqqk-14/?versions&max-keys=999 2024-04-09T21:59:23.793 DEBUG:boto:Data: 2024-04-09T21:59:23.793 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.793 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.793 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.793 DEBUG:boto:Params: {} 2024-04-09T21:59:23.793 DEBUG:boto:Token: None 2024-04-09T21:59:23.793 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/?versions 2024-04-09T21:59:23.794 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ= 2024-04-09T21:59:23.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M6ATPomMz0y5IsBx8m3l/hU4XKQ=', 'Content-Length': '0'} 2024-04-09T21:59:23.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de0106a2b2d59b5-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.796 DEBUG:boto:b'xryqqk-14999falsedummyepEi8oOzCDvqM-UXxuGeMGtcp84wSRLtrue2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1hdIPgRU-toIRPGjo3YehLHo-5mxTZ.strue2024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1true2024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:23.796 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.797 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.797 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.797 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.797 DEBUG:boto:Method: GET 2024-04-09T21:59:23.797 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.797 DEBUG:boto:Data: 2024-04-09T21:59:23.797 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.797 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.797 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.797 DEBUG:boto:Params: {} 2024-04-09T21:59:23.797 DEBUG:boto:Token: None 2024-04-09T21:59:23.798 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.798 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E= 2024-04-09T21:59:23.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E=', 'Content-Length': '0'} 2024-04-09T21:59:23.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000019943b493f301ba2-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.800 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.800 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.800 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.800 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.800 DEBUG:boto:Method: GET 2024-04-09T21:59:23.800 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.800 DEBUG:boto:Data: 2024-04-09T21:59:23.800 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.800 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.800 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.800 DEBUG:boto:Params: {} 2024-04-09T21:59:23.800 DEBUG:boto:Token: None 2024-04-09T21:59:23.801 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.801 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E= 2024-04-09T21:59:23.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E=', 'Content-Length': '0'} 2024-04-09T21:59:23.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008b359459a86908b4-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.803 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.803 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.803 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.803 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.803 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.803 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.803 DEBUG:boto:Data: 2024-04-09T21:59:23.803 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.803 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.803 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.803 DEBUG:boto:Params: {} 2024-04-09T21:59:23.804 DEBUG:boto:Token: None 2024-04-09T21:59:23.804 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:W3d+addbeoK0sRgt5cHZlMtbElg= 2024-04-09T21:59:23.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:W3d+addbeoK0sRgt5cHZlMtbElg=', 'Content-Length': '0'} 2024-04-09T21:59:23.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004be02412a1413945-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.805 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.805 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.806 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.806 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.806 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.806 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.806 DEBUG:boto:Data: 2024-04-09T21:59:23.806 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.806 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.806 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.806 DEBUG:boto:Params: {} 2024-04-09T21:59:23.806 DEBUG:boto:Token: None 2024-04-09T21:59:23.806 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.807 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:W3d+addbeoK0sRgt5cHZlMtbElg= 2024-04-09T21:59:23.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:W3d+addbeoK0sRgt5cHZlMtbElg=', 'Content-Length': '0'} 2024-04-09T21:59:23.808 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000033e2f843a4a70687-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:23.809 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.809 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.809 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.809 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.809 DEBUG:boto:Method: GET 2024-04-09T21:59:23.809 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.809 DEBUG:boto:Data: 2024-04-09T21:59:23.809 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.809 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.809 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.809 DEBUG:boto:Params: {} 2024-04-09T21:59:23.810 DEBUG:boto:Token: None 2024-04-09T21:59:23.810 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.810 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E= 2024-04-09T21:59:23.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E=', 'Content-Length': '0'} 2024-04-09T21:59:23.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dad2eb15af47a74e-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.812 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.812 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.812 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.812 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.812 DEBUG:boto:Method: GET 2024-04-09T21:59:23.812 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.812 DEBUG:boto:Data: 2024-04-09T21:59:23.812 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.812 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.812 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.812 DEBUG:boto:Params: {} 2024-04-09T21:59:23.813 DEBUG:boto:Token: None 2024-04-09T21:59:23.813 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.813 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E= 2024-04-09T21:59:23.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E=', 'Content-Length': '0'} 2024-04-09T21:59:23.817 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059734008d5167ff3-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.818 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.818 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.818 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.818 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.818 DEBUG:boto:Method: GET 2024-04-09T21:59:23.818 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.818 DEBUG:boto:Data: 2024-04-09T21:59:23.818 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.818 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.818 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.819 DEBUG:boto:Params: {} 2024-04-09T21:59:23.819 DEBUG:boto:Token: None 2024-04-09T21:59:23.819 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.819 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc= 2024-04-09T21:59:23.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc=', 'Content-Length': '0'} 2024-04-09T21:59:23.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080e3a121bd4491d-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.862 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.863 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.863 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.863 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.863 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.863 DEBUG:boto:Method: GET 2024-04-09T21:59:23.863 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.863 DEBUG:boto:Data: 2024-04-09T21:59:23.863 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.863 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.863 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.863 DEBUG:boto:Params: {} 2024-04-09T21:59:23.864 DEBUG:boto:Token: None 2024-04-09T21:59:23.864 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.864 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc= 2024-04-09T21:59:23.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc=', 'Content-Length': '0'} 2024-04-09T21:59:23.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000999ee1ceb57c9bf4-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.866 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.867 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.867 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.867 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.867 DEBUG:boto:Path: /xryqqk-14/dummy 2024-04-09T21:59:23.867 DEBUG:boto:Data: 2024-04-09T21:59:23.867 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.867 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.867 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.868 DEBUG:boto:Params: {} 2024-04-09T21:59:23.868 DEBUG:boto:Token: None 2024-04-09T21:59:23.868 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy 2024-04-09T21:59:23.868 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uxpA4iTJKycuwy4ElPDG7U4PV4o= 2024-04-09T21:59:23.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uxpA4iTJKycuwy4ElPDG7U4PV4o=', 'Content-Length': '0'} 2024-04-09T21:59:23.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b22660d722bcc7a-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.870 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.870 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.870 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.870 DEBUG:boto:Path: /xryqqk-14/dummy 2024-04-09T21:59:23.870 DEBUG:boto:Data: 2024-04-09T21:59:23.870 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.870 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.871 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.871 DEBUG:boto:Params: {} 2024-04-09T21:59:23.871 DEBUG:boto:Token: None 2024-04-09T21:59:23.871 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy 2024-04-09T21:59:23.871 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uxpA4iTJKycuwy4ElPDG7U4PV4o= 2024-04-09T21:59:23.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uxpA4iTJKycuwy4ElPDG7U4PV4o=', 'Content-Length': '0'} 2024-04-09T21:59:23.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000662e12cf9cf34f88-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:23.873 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.873 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.873 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.873 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.873 DEBUG:boto:Method: GET 2024-04-09T21:59:23.873 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.874 DEBUG:boto:Data: 2024-04-09T21:59:23.874 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.874 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.874 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.874 DEBUG:boto:Params: {} 2024-04-09T21:59:23.874 DEBUG:boto:Token: None 2024-04-09T21:59:23.874 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.874 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E= 2024-04-09T21:59:23.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E=', 'Content-Length': '0'} 2024-04-09T21:59:23.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096c0f707d232e67d-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.876 DEBUG:boto:path=/xryqqk-14/dummy 2024-04-09T21:59:23.876 DEBUG:boto:auth_path=/xryqqk-14/dummy 2024-04-09T21:59:23.876 DEBUG:boto:path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.876 DEBUG:boto:auth_path=/xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.877 DEBUG:boto:Method: GET 2024-04-09T21:59:23.877 DEBUG:boto:Path: /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.877 DEBUG:boto:Data: 2024-04-09T21:59:23.877 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.877 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.877 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.877 DEBUG:boto:Params: {} 2024-04-09T21:59:23.877 DEBUG:boto:Token: None 2024-04-09T21:59:23.877 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/dummy?versionId=epEi8oOzCDvqM-UXxuGeMGtcp84wSRL 2024-04-09T21:59:23.877 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E= 2024-04-09T21:59:23.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y1IP8+KC4U5HBGw+081pivpkZ4E=', 'Content-Length': '0'} 2024-04-09T21:59:23.879 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:00 GMT'), ('x-amz-version-id', 'epEi8oOzCDvqM-UXxuGeMGtcp84wSRL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e4dbe232ac52f6fb-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.879 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.879 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.879 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.879 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.879 DEBUG:boto:Method: GET 2024-04-09T21:59:23.879 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.880 DEBUG:boto:Data: 2024-04-09T21:59:23.880 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.880 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.880 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.880 DEBUG:boto:Params: {} 2024-04-09T21:59:23.880 DEBUG:boto:Token: None 2024-04-09T21:59:23.880 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.880 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc= 2024-04-09T21:59:23.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc=', 'Content-Length': '0'} 2024-04-09T21:59:23.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ca5950f032528f6-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.883 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.883 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.883 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.883 DEBUG:boto:path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.883 DEBUG:boto:auth_path=/xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.883 DEBUG:boto:Method: GET 2024-04-09T21:59:23.883 DEBUG:boto:Path: /xryqqk-14/?prefix=dummy 2024-04-09T21:59:23.883 DEBUG:boto:Data: 2024-04-09T21:59:23.883 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.884 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.884 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.884 DEBUG:boto:Params: {} 2024-04-09T21:59:23.884 DEBUG:boto:Token: None 2024-04-09T21:59:23.884 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.884 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc= 2024-04-09T21:59:23.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc=', 'Content-Length': '0'} 2024-04-09T21:59:23.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534838a687deefa6-006615ba3b-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.886 DEBUG:boto:b'xryqqk-14dummy1000falsedummy2024-04-09T21:58:00.598Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:23.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:23.887 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.887 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.887 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.887 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.887 DEBUG:boto:Method: GET 2024-04-09T21:59:23.887 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.887 DEBUG:boto:Data: 2024-04-09T21:59:23.888 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.888 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.888 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.888 DEBUG:boto:Params: {} 2024-04-09T21:59:23.888 DEBUG:boto:Token: None 2024-04-09T21:59:23.888 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.888 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw= 2024-04-09T21:59:23.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw=', 'Content-Length': '0'} 2024-04-09T21:59:23.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c37521dc7032b691-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.890 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.890 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.890 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.890 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.891 DEBUG:boto:Method: GET 2024-04-09T21:59:23.891 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.891 DEBUG:boto:Data: 2024-04-09T21:59:23.891 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.891 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.891 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.891 DEBUG:boto:Params: {} 2024-04-09T21:59:23.891 DEBUG:boto:Token: None 2024-04-09T21:59:23.891 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.891 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw= 2024-04-09T21:59:23.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw=', 'Content-Length': '0'} 2024-04-09T21:59:23.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009583af6d1ad7e169-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.934 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.934 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.935 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.935 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.935 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.935 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.935 DEBUG:boto:Data: 2024-04-09T21:59:23.935 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.935 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.935 DEBUG:boto:Params: {} 2024-04-09T21:59:23.935 DEBUG:boto:Token: None 2024-04-09T21:59:23.935 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ojYmr3SGmdTdWPdOfmy8I0LcPtw= 2024-04-09T21:59:23.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ojYmr3SGmdTdWPdOfmy8I0LcPtw=', 'Content-Length': '0'} 2024-04-09T21:59:23.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7328ecef3b491a6-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.937 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.938 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.938 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.938 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.938 DEBUG:boto:Method: HEAD 2024-04-09T21:59:23.938 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.938 DEBUG:boto:Data: 2024-04-09T21:59:23.938 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.938 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.938 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.938 DEBUG:boto:Params: {} 2024-04-09T21:59:23.938 DEBUG:boto:Token: None 2024-04-09T21:59:23.939 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ojYmr3SGmdTdWPdOfmy8I0LcPtw= 2024-04-09T21:59:23.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ojYmr3SGmdTdWPdOfmy8I0LcPtw=', 'Content-Length': '0'} 2024-04-09T21:59:23.940 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca2c5af423608665-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:23.941 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.941 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.941 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.941 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.941 DEBUG:boto:Method: GET 2024-04-09T21:59:23.941 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.941 DEBUG:boto:Data: 2024-04-09T21:59:23.942 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.942 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.942 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.942 DEBUG:boto:Params: {} 2024-04-09T21:59:23.942 DEBUG:boto:Token: None 2024-04-09T21:59:23.942 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.942 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw= 2024-04-09T21:59:23.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw=', 'Content-Length': '0'} 2024-04-09T21:59:23.944 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000041af0b1f0f4b5dee-006615ba3b-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.990 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.990 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:23.990 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.991 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.991 DEBUG:boto:Method: GET 2024-04-09T21:59:23.991 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.991 DEBUG:boto:Data: 2024-04-09T21:59:23.991 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.991 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:23.991 DEBUG:boto:Port: 8000 2024-04-09T21:59:23.991 DEBUG:boto:Params: {} 2024-04-09T21:59:23.991 DEBUG:boto:Token: None 2024-04-09T21:59:23.991 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:23.992 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw= 2024-04-09T21:59:23.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xFB/Xk1GUjEKceXorvVhJ/Tbeqw=', 'Content-Length': '0'} 2024-04-09T21:59:23.993 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ffa09e8d159ccdc-006615ba3b-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:23.993 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:23.993 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:23.993 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:23.994 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:23.994 DEBUG:boto:Method: GET 2024-04-09T21:59:23.994 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:23.994 DEBUG:boto:Data: 2024-04-09T21:59:23.994 DEBUG:boto:Headers: {} 2024-04-09T21:59:23.994 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:23.994 DEBUG:boto:Port: 8001 2024-04-09T21:59:23.994 DEBUG:boto:Params: {} 2024-04-09T21:59:23.994 DEBUG:boto:Token: None 2024-04-09T21:59:23.994 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:23 GMT /xryqqk-14/ 2024-04-09T21:59:23.995 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc= 2024-04-09T21:59:23.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Hbcf1u+/PhdE4+nNj5VvMI5urnc=', 'Content-Length': '0'} 2024-04-09T21:59:23.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bdd3e28c4fe4c3-006615ba3b-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.038 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.039 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.039 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.039 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.039 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.039 DEBUG:boto:Method: GET 2024-04-09T21:59:24.039 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.039 DEBUG:boto:Data: 2024-04-09T21:59:24.039 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.040 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.040 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.040 DEBUG:boto:Params: {} 2024-04-09T21:59:24.040 DEBUG:boto:Token: None 2024-04-09T21:59:24.040 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.040 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b96b1761660544c-006615ba3c-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.042 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.042 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.042 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.043 DEBUG:boto:Method: HEAD 2024-04-09T21:59:24.043 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:59:24.043 DEBUG:boto:Data: 2024-04-09T21:59:24.043 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.043 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.043 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.043 DEBUG:boto:Params: {} 2024-04-09T21:59:24.043 DEBUG:boto:Token: None 2024-04-09T21:59:24.043 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a1 2024-04-09T21:59:24.043 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Oa4ff6aZFONgQkYmStnr/lsSZMw= 2024-04-09T21:59:24.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Oa4ff6aZFONgQkYmStnr/lsSZMw=', 'Content-Length': '0'} 2024-04-09T21:59:24.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2c97b72a86d015b-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.045 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.045 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.046 DEBUG:boto:Method: HEAD 2024-04-09T21:59:24.046 DEBUG:boto:Path: /xryqqk-14/obj-a1 2024-04-09T21:59:24.046 DEBUG:boto:Data: 2024-04-09T21:59:24.046 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.046 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.046 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.046 DEBUG:boto:Params: {} 2024-04-09T21:59:24.046 DEBUG:boto:Token: None 2024-04-09T21:59:24.046 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a1 2024-04-09T21:59:24.047 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Oa4ff6aZFONgQkYmStnr/lsSZMw= 2024-04-09T21:59:24.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Oa4ff6aZFONgQkYmStnr/lsSZMw=', 'Content-Length': '0'} 2024-04-09T21:59:24.048 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8ad411eea6a6a2b-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-09T21:59:24.049 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.049 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.049 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.049 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.049 DEBUG:boto:Method: GET 2024-04-09T21:59:24.049 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.049 DEBUG:boto:Data: 2024-04-09T21:59:24.049 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.049 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.049 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.049 DEBUG:boto:Params: {} 2024-04-09T21:59:24.050 DEBUG:boto:Token: None 2024-04-09T21:59:24.050 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.050 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KepKEciyrWlXtEsFxGVsi37Msi4= 2024-04-09T21:59:24.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KepKEciyrWlXtEsFxGVsi37Msi4=', 'Content-Length': '0'} 2024-04-09T21:59:24.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a47568bb9ea1c14-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.094 DEBUG:boto:path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.094 DEBUG:boto:auth_path=/xryqqk-14/obj-a1 2024-04-09T21:59:24.094 DEBUG:boto:path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.095 DEBUG:boto:auth_path=/xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.095 DEBUG:boto:Method: GET 2024-04-09T21:59:24.095 DEBUG:boto:Path: /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.095 DEBUG:boto:Data: 2024-04-09T21:59:24.095 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.095 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.095 DEBUG:boto:Params: {} 2024-04-09T21:59:24.095 DEBUG:boto:Token: None 2024-04-09T21:59:24.095 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a1?versionId=hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s 2024-04-09T21:59:24.095 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KepKEciyrWlXtEsFxGVsi37Msi4= 2024-04-09T21:59:24.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KepKEciyrWlXtEsFxGVsi37Msi4=', 'Content-Length': '0'} 2024-04-09T21:59:24.097 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:20 GMT'), ('x-amz-version-id', 'hdIPgRU-toIRPGjo3YehLHo-5mxTZ.s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098c00abd6d047b80-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.097 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.098 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.098 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.098 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.098 DEBUG:boto:Method: GET 2024-04-09T21:59:24.098 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.098 DEBUG:boto:Data: 2024-04-09T21:59:24.098 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.098 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.098 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.098 DEBUG:boto:Params: {} 2024-04-09T21:59:24.098 DEBUG:boto:Token: None 2024-04-09T21:59:24.099 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.099 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000239b10452426aac3-006615ba3c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.101 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.101 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.101 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.101 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.101 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.101 DEBUG:boto:Method: GET 2024-04-09T21:59:24.102 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a1 2024-04-09T21:59:24.102 DEBUG:boto:Data: 2024-04-09T21:59:24.102 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.102 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.102 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.102 DEBUG:boto:Params: {} 2024-04-09T21:59:24.102 DEBUG:boto:Token: None 2024-04-09T21:59:24.102 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.102 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f02e2f010f71ce4-006615ba3c-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.105 DEBUG:boto:b'xryqqk-14obj-a11000falseobj-a12024-04-09T21:58:20.993Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:24.105 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.105 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.105 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.105 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.105 DEBUG:boto:Method: GET 2024-04-09T21:59:24.105 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.105 DEBUG:boto:Data: 2024-04-09T21:59:24.106 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.106 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.106 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.106 DEBUG:boto:Params: {} 2024-04-09T21:59:24.106 DEBUG:boto:Token: None 2024-04-09T21:59:24.106 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.106 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8= 2024-04-09T21:59:24.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8=', 'Content-Length': '0'} 2024-04-09T21:59:24.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000570df129cc719408-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.108 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.108 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.108 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.108 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.109 DEBUG:boto:Method: GET 2024-04-09T21:59:24.109 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.109 DEBUG:boto:Data: 2024-04-09T21:59:24.109 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.109 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.109 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.109 DEBUG:boto:Params: {} 2024-04-09T21:59:24.109 DEBUG:boto:Token: None 2024-04-09T21:59:24.109 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.109 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8= 2024-04-09T21:59:24.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8=', 'Content-Length': '0'} 2024-04-09T21:59:24.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000011aaa880ef8f0c0f-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.111 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.111 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.111 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.111 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.111 DEBUG:boto:Method: HEAD 2024-04-09T21:59:24.112 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.112 DEBUG:boto:Data: 2024-04-09T21:59:24.112 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.112 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.112 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.112 DEBUG:boto:Params: {} 2024-04-09T21:59:24.112 DEBUG:boto:Token: None 2024-04-09T21:59:24.112 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.112 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+gBw4Nra7AKd2xyAvgxqPMWWsa0= 2024-04-09T21:59:24.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+gBw4Nra7AKd2xyAvgxqPMWWsa0=', 'Content-Length': '0'} 2024-04-09T21:59:24.114 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007a90a92df4653742-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.114 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.114 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.114 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.114 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.114 DEBUG:boto:Method: HEAD 2024-04-09T21:59:24.114 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.115 DEBUG:boto:Data: 2024-04-09T21:59:24.115 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.115 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.115 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.115 DEBUG:boto:Params: {} 2024-04-09T21:59:24.115 DEBUG:boto:Token: None 2024-04-09T21:59:24.115 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.115 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+gBw4Nra7AKd2xyAvgxqPMWWsa0= 2024-04-09T21:59:24.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+gBw4Nra7AKd2xyAvgxqPMWWsa0=', 'Content-Length': '0'} 2024-04-09T21:59:24.117 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091348f2c5e7788d9-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:24.117 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.117 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.117 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.117 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.117 DEBUG:boto:Method: GET 2024-04-09T21:59:24.117 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.117 DEBUG:boto:Data: 2024-04-09T21:59:24.118 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.118 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.118 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.118 DEBUG:boto:Params: {} 2024-04-09T21:59:24.118 DEBUG:boto:Token: None 2024-04-09T21:59:24.118 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.118 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8= 2024-04-09T21:59:24.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8=', 'Content-Length': '0'} 2024-04-09T21:59:24.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000632253b1b8c1b3bc-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.120 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.120 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.120 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.120 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.120 DEBUG:boto:Method: GET 2024-04-09T21:59:24.120 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.120 DEBUG:boto:Data: 2024-04-09T21:59:24.120 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.120 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.121 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.121 DEBUG:boto:Params: {} 2024-04-09T21:59:24.121 DEBUG:boto:Token: None 2024-04-09T21:59:24.121 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.121 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8= 2024-04-09T21:59:24.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8=', 'Content-Length': '0'} 2024-04-09T21:59:24.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b333173771a8e18b-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.166 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.167 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.167 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.167 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.167 DEBUG:boto:Method: GET 2024-04-09T21:59:24.167 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.167 DEBUG:boto:Data: 2024-04-09T21:59:24.167 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.167 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.167 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.167 DEBUG:boto:Params: {} 2024-04-09T21:59:24.168 DEBUG:boto:Token: None 2024-04-09T21:59:24.168 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.168 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1e316b1404aa11-006615ba3c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.170 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.171 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.171 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.171 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.171 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.171 DEBUG:boto:Method: GET 2024-04-09T21:59:24.171 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.171 DEBUG:boto:Data: 2024-04-09T21:59:24.171 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.171 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.171 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.171 DEBUG:boto:Params: {} 2024-04-09T21:59:24.172 DEBUG:boto:Token: None 2024-04-09T21:59:24.172 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.172 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe10ddeb9273c0f6-006615ba3c-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.218 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.219 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.219 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.219 DEBUG:boto:Method: HEAD 2024-04-09T21:59:24.219 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:59:24.219 DEBUG:boto:Data: 2024-04-09T21:59:24.219 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.219 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.219 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.219 DEBUG:boto:Params: {} 2024-04-09T21:59:24.220 DEBUG:boto:Token: None 2024-04-09T21:59:24.220 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2 2024-04-09T21:59:24.220 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:A5axEJEc9g6ME+8t1przeU2bYuk= 2024-04-09T21:59:24.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A5axEJEc9g6ME+8t1przeU2bYuk=', 'Content-Length': '0'} 2024-04-09T21:59:24.222 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dd91889a84fbffeb-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.222 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.222 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.222 DEBUG:boto:Method: HEAD 2024-04-09T21:59:24.225 DEBUG:boto:Path: /xryqqk-14/obj-a2 2024-04-09T21:59:24.225 DEBUG:boto:Data: 2024-04-09T21:59:24.226 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.226 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.226 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.226 DEBUG:boto:Params: {} 2024-04-09T21:59:24.226 DEBUG:boto:Token: None 2024-04-09T21:59:24.226 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2 2024-04-09T21:59:24.226 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:A5axEJEc9g6ME+8t1przeU2bYuk= 2024-04-09T21:59:24.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A5axEJEc9g6ME+8t1przeU2bYuk=', 'Content-Length': '0'} 2024-04-09T21:59:24.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d68d00eeac3ffea6-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-09T21:59:24.228 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.229 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.229 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.229 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.229 DEBUG:boto:Method: GET 2024-04-09T21:59:24.229 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.229 DEBUG:boto:Data: 2024-04-09T21:59:24.229 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.229 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.229 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.229 DEBUG:boto:Params: {} 2024-04-09T21:59:24.229 DEBUG:boto:Token: None 2024-04-09T21:59:24.230 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.230 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8= 2024-04-09T21:59:24.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8=', 'Content-Length': '0'} 2024-04-09T21:59:24.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007692ca9ce41eaa40-006615ba3c-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.274 DEBUG:boto:path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.275 DEBUG:boto:auth_path=/xryqqk-14/obj-a2 2024-04-09T21:59:24.275 DEBUG:boto:path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.275 DEBUG:boto:auth_path=/xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.275 DEBUG:boto:Method: GET 2024-04-09T21:59:24.275 DEBUG:boto:Path: /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.275 DEBUG:boto:Data: 2024-04-09T21:59:24.275 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.275 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.275 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.275 DEBUG:boto:Params: {} 2024-04-09T21:59:24.275 DEBUG:boto:Token: None 2024-04-09T21:59:24.276 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/obj-a2?versionId=WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1 2024-04-09T21:59:24.276 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8= 2024-04-09T21:59:24.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R1tsNdehALQEwphr7/5ulamMAG8=', 'Content-Length': '0'} 2024-04-09T21:59:24.277 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:58:21 GMT'), ('x-amz-version-id', 'WF55iVYhoNSnsJfIDx9xCUvmUCgqsP1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-14:a5723776-f142-428c-a6df-d7bf949257da.4372.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077ba6e03023118eb-006615ba3c-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.318 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.318 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.319 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.319 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.319 DEBUG:boto:Method: GET 2024-04-09T21:59:24.319 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.319 DEBUG:boto:Data: 2024-04-09T21:59:24.319 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.319 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:24.319 DEBUG:boto:Port: 8001 2024-04-09T21:59:24.319 DEBUG:boto:Params: {} 2024-04-09T21:59:24.319 DEBUG:boto:Token: None 2024-04-09T21:59:24.319 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d250cfe525c1af69-006615ba3c-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.322 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.322 DEBUG:boto:path=/xryqqk-14/ 2024-04-09T21:59:24.323 DEBUG:boto:auth_path=/xryqqk-14/ 2024-04-09T21:59:24.323 DEBUG:boto:path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.323 DEBUG:boto:auth_path=/xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.323 DEBUG:boto:Method: GET 2024-04-09T21:59:24.323 DEBUG:boto:Path: /xryqqk-14/?prefix=obj-a2 2024-04-09T21:59:24.323 DEBUG:boto:Data: 2024-04-09T21:59:24.323 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.323 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.323 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.323 DEBUG:boto:Params: {} 2024-04-09T21:59:24.323 DEBUG:boto:Token: None 2024-04-09T21:59:24.323 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-14/ 2024-04-09T21:59:24.324 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc= 2024-04-09T21:59:24.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VuXLyxbTXZ1NERvk71AdDL5dBnc=', 'Content-Length': '0'} 2024-04-09T21:59:24.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a025c3499ddd097-006615ba3c-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.326 DEBUG:boto:b'xryqqk-14obj-a21000falseobj-a22024-04-09T21:58:21.087Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:24.326 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-14 zones={a2, a1} 2024-04-09T21:59:24.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-04-09T21:59:24.328 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:24.328 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:24.328 DEBUG:boto:Method: PUT 2024-04-09T21:59:24.329 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T21:59:24.329 DEBUG:boto:Data: 2024-04-09T21:59:24.329 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.329 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.329 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.329 DEBUG:boto:Params: {} 2024-04-09T21:59:24.329 DEBUG:boto:Token: None 2024-04-09T21:59:24.329 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-15/ 2024-04-09T21:59:24.329 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yOqKSNhzFGWY1LKX0W/itkAiJ2M= 2024-04-09T21:59:24.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yOqKSNhzFGWY1LKX0W/itkAiJ2M=', 'Content-Length': '0'} 2024-04-09T21:59:24.343 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000969be771d082a2fb-006615ba3c-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.343 DEBUG:rgw_multi.tests:created bucket=xryqqk-15 2024-04-09T21:59:24.344 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:24.344 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:24.344 DEBUG:boto:path=/xryqqk-15/?versioning 2024-04-09T21:59:24.344 DEBUG:boto:auth_path=/xryqqk-15/?versioning 2024-04-09T21:59:24.344 DEBUG:boto:Method: PUT 2024-04-09T21:59:24.344 DEBUG:boto:Path: /xryqqk-15/?versioning 2024-04-09T21:59:24.344 DEBUG:boto:Data: Enabled Disabled 2024-04-09T21:59:24.344 DEBUG:boto:Headers: {} 2024-04-09T21:59:24.344 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:24.344 DEBUG:boto:Port: 8000 2024-04-09T21:59:24.345 DEBUG:boto:Params: {} 2024-04-09T21:59:24.345 DEBUG:boto:Token: None 2024-04-09T21:59:24.345 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 21:59:24 GMT /xryqqk-15/?versioning 2024-04-09T21:59:24.345 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PdD/O/p7G06M3U2mgSflwe1cfVU= 2024-04-09T21:59:24.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PdD/O/p7G06M3U2mgSflwe1cfVU=', 'Content-Length': '228'} 2024-04-09T21:59:24.350 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000095284c463cb0846b-006615ba3c-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:24.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:59:24.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:59:24.350 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:59:24.401 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:24.401 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:24.435 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T21:59:24.448 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.449 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699964.349473_220.1", 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:59:24.349473Z" 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:24.450 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T21:59:24.451 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699964.349473_220.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:59:24.451 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T21:59:24.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:59:24.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:59:24.452 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:59:24.501 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:24.502 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:24.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:24.534+0000 7fc2bd833780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:59:24.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:24.535+0000 7fc2bd833780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:59:24.550 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699964.336118_215.1", 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:59:24.336118Z", 2024-04-09T21:59:24.551 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:24.552 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:59:24Z" 2024-04-09T21:59:24.553 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:59:24.554 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699964.336118_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:59:24.336118Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:59:24Z" } 2024-04-09T21:59:24.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699964.336118_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:59:24.336118Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:59:24.554 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699964.349473_220.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:59:24.555 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699964.336118_215.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:59:24.555 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712699964.349473_220.1 target=1_1712699964.336118_215.1 2024-04-09T21:59:29.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:59:29.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T21:59:29.556 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T21:59:29.614 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:29.614 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:29.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:29.648+0000 7f21ca0fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:59:29.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:29.649+0000 7f21ca0fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T21:59:29.663 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:29.664 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:29.665 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699964.349473_220.1", 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:59:29.666 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:59:24.349473Z", 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T21:59:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T21:59:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T21:59:29Z" 2024-04-09T21:59:29.669 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T21:59:29.669 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712699964.349473_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:59:24.349473Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T21:59:29Z" } 2024-04-09T21:59:29.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699964.349473_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:59:24.349473Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T21:59:29.670 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699964.349473_220.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:59:29.670 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699964.349473_220.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T21:59:29.670 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T21:59:29.670 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:29.670 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:29.670 DEBUG:boto:Method: HEAD 2024-04-09T21:59:29.671 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T21:59:29.671 DEBUG:boto:Data: 2024-04-09T21:59:29.671 DEBUG:boto:Headers: {} 2024-04-09T21:59:29.671 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:29.671 DEBUG:boto:Port: 8000 2024-04-09T21:59:29.671 DEBUG:boto:Params: {} 2024-04-09T21:59:29.671 DEBUG:boto:Token: None 2024-04-09T21:59:29.671 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:29 GMT /xryqqk-15/ 2024-04-09T21:59:29.671 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PgjWY/ZujvxblR/dpL+LX7nXCrU= 2024-04-09T21:59:29.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:29 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PgjWY/ZujvxblR/dpL+LX7nXCrU=', 'Content-Length': '0'} 2024-04-09T21:59:29.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091aaad06f1e2f9fb-006615ba41-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:29.674 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:29.674 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:29.674 DEBUG:boto:Method: PUT 2024-04-09T21:59:29.674 DEBUG:boto:Path: /xryqqk-15/dummy 2024-04-09T21:59:29.675 DEBUG:boto:Data: 2024-04-09T21:59:29.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T21:59:29.675 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:29.675 DEBUG:boto:Port: 8000 2024-04-09T21:59:29.675 DEBUG:boto:Params: {} 2024-04-09T21:59:29.675 DEBUG:boto:Token: None 2024-04-09T21:59:29.675 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 21:59:29 GMT /xryqqk-15/dummy 2024-04-09T21:59:29.675 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ysxSQczjH1thZZubHZtPL+x+vRU= 2024-04-09T21:59:29.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:59:29 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ysxSQczjH1thZZubHZtPL+x+vRU='} 2024-04-09T21:59:29.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-amz-request-id', 'tx00000cd4da2b412b6871b-006615ba41-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:29.686 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-15 2024-04-09T21:59:29.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:29.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:29.687 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:29.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:29.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:29.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:29.777+0000 7fad92ccb780 1 waiting to reach incremental sync.. 2024-04-09T21:59:34.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:34.778+0000 7fad92ccb780 1 waiting to reach incremental sync.. 2024-04-09T21:59:39.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:39.780+0000 7fad92ccb780 1 bucket sync caught up with source: 2024-04-09T21:59:39.781 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.7.11, , , , , ] 2024-04-09T21:59:39.781 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.7.11, , , , , ] 2024-04-09T21:59:39.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:39.780+0000 7fad92ccb780 0 bucket checkpoint complete 2024-04-09T21:59:39.798 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-15 2024-04-09T21:59:39.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:39.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T21:59:39.798 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T21:59:39.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T21:59:39.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T21:59:39.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:39.902+0000 7ffbb6585780 1 waiting to reach incremental sync.. 2024-04-09T21:59:44.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:44.902+0000 7ffbb6585780 1 waiting to reach incremental sync.. 2024-04-09T21:59:49.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:49.903+0000 7ffbb6585780 1 waiting to reach incremental sync.. 2024-04-09T21:59:54.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:54.904+0000 7ffbb6585780 1 waiting to reach incremental sync.. 2024-04-09T21:59:59.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:59.906+0000 7ffbb6585780 1 bucket sync caught up with source: 2024-04-09T21:59:59.906 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2024-04-09T21:59:59.907 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.11.11, , , , , ] 2024-04-09T21:59:59.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T21:59:59.906+0000 7ffbb6585780 0 bucket checkpoint complete 2024-04-09T21:59:59.923 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-15 zones={a2, a1} 2024-04-09T21:59:59.923 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.923 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.923 DEBUG:boto:Method: HEAD 2024-04-09T21:59:59.923 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T21:59:59.923 DEBUG:boto:Data: 2024-04-09T21:59:59.924 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.924 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.924 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.924 DEBUG:boto:Params: {} 2024-04-09T21:59:59.924 DEBUG:boto:Token: None 2024-04-09T21:59:59.924 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/ 2024-04-09T21:59:59.924 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mjHPZngb48VEgJclZ2VKM/h6KK0= 2024-04-09T21:59:59.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mjHPZngb48VEgJclZ2VKM/h6KK0=', 'Content-Length': '0'} 2024-04-09T21:59:59.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075a13e808399938b-006615ba5f-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.929 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.929 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.929 DEBUG:boto:Method: HEAD 2024-04-09T21:59:59.929 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T21:59:59.929 DEBUG:boto:Data: 2024-04-09T21:59:59.929 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.929 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.929 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.930 DEBUG:boto:Params: {} 2024-04-09T21:59:59.930 DEBUG:boto:Token: None 2024-04-09T21:59:59.930 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/ 2024-04-09T21:59:59.930 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mjHPZngb48VEgJclZ2VKM/h6KK0= 2024-04-09T21:59:59.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mjHPZngb48VEgJclZ2VKM/h6KK0=', 'Content-Length': '0'} 2024-04-09T21:59:59.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063c5ec0f9f03d990-006615ba5f-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.932 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T21:59:59.933 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.933 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.933 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.933 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.933 DEBUG:boto:Method: GET 2024-04-09T21:59:59.933 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.933 DEBUG:boto:Data: 2024-04-09T21:59:59.933 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.933 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.933 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.933 DEBUG:boto:Params: {} 2024-04-09T21:59:59.934 DEBUG:boto:Token: None 2024-04-09T21:59:59.934 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/?versions 2024-04-09T21:59:59.934 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE= 2024-04-09T21:59:59.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE=', 'Content-Length': '0'} 2024-04-09T21:59:59.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba8433d590c008b8-006615ba5f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.936 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:59.937 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:59.937 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T21:59:59.937 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.937 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.938 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.938 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.938 DEBUG:boto:Method: GET 2024-04-09T21:59:59.938 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.938 DEBUG:boto:Data: 2024-04-09T21:59:59.938 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.938 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.938 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.938 DEBUG:boto:Params: {} 2024-04-09T21:59:59.938 DEBUG:boto:Token: None 2024-04-09T21:59:59.939 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/?versions 2024-04-09T21:59:59.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE= 2024-04-09T21:59:59.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE=', 'Content-Length': '0'} 2024-04-09T21:59:59.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99bb37613dab1ce-006615ba5f-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.941 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:59.942 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T21:59:59.942 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.942 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.942 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.942 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.942 DEBUG:boto:Method: GET 2024-04-09T21:59:59.943 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.943 DEBUG:boto:Data: 2024-04-09T21:59:59.943 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.943 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.943 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.943 DEBUG:boto:Params: {} 2024-04-09T21:59:59.943 DEBUG:boto:Token: None 2024-04-09T21:59:59.943 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/?versions 2024-04-09T21:59:59.943 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE= 2024-04-09T21:59:59.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE=', 'Content-Length': '0'} 2024-04-09T21:59:59.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001af02790adae55a-006615ba5f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.946 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:59.947 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.947 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.947 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.947 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.947 DEBUG:boto:Method: GET 2024-04-09T21:59:59.947 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T21:59:59.947 DEBUG:boto:Data: 2024-04-09T21:59:59.947 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.947 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.947 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.948 DEBUG:boto:Params: {} 2024-04-09T21:59:59.948 DEBUG:boto:Token: None 2024-04-09T21:59:59.948 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/?versions 2024-04-09T21:59:59.948 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE= 2024-04-09T21:59:59.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j8BSGvBMWXgkFszi+R0QwfupxHE=', 'Content-Length': '0'} 2024-04-09T21:59:59.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ad76b50fe31e2c8-006615ba5f-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.950 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:59.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:59.951 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.951 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.951 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.951 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.951 DEBUG:boto:Method: GET 2024-04-09T21:59:59.951 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.951 DEBUG:boto:Data: 2024-04-09T21:59:59.951 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.951 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.951 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.952 DEBUG:boto:Params: {} 2024-04-09T21:59:59.952 DEBUG:boto:Token: None 2024-04-09T21:59:59.952 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.952 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8= 2024-04-09T21:59:59.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8=', 'Content-Length': '0'} 2024-04-09T21:59:59.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000be445ffe7ce5cb6b-006615ba5f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.954 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.954 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.954 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.954 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.955 DEBUG:boto:Method: GET 2024-04-09T21:59:59.955 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.955 DEBUG:boto:Data: 2024-04-09T21:59:59.955 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.955 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.955 DEBUG:boto:Params: {} 2024-04-09T21:59:59.955 DEBUG:boto:Token: None 2024-04-09T21:59:59.955 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.955 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8= 2024-04-09T21:59:59.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8=', 'Content-Length': '0'} 2024-04-09T21:59:59.957 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000519fa4b1496dca21-006615ba5f-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.957 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.957 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.958 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.958 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.958 DEBUG:boto:Method: HEAD 2024-04-09T21:59:59.958 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.958 DEBUG:boto:Data: 2024-04-09T21:59:59.958 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.958 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.958 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.958 DEBUG:boto:Params: {} 2024-04-09T21:59:59.958 DEBUG:boto:Token: None 2024-04-09T21:59:59.959 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.959 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aMTyHohDjfz8o7IqTWZ1eiEclrQ= 2024-04-09T21:59:59.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aMTyHohDjfz8o7IqTWZ1eiEclrQ=', 'Content-Length': '0'} 2024-04-09T21:59:59.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad108d84ff36b627-006615ba5f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.961 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.961 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.961 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.961 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.962 DEBUG:boto:Method: HEAD 2024-04-09T21:59:59.962 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.962 DEBUG:boto:Data: 2024-04-09T21:59:59.962 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.962 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.962 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.962 DEBUG:boto:Params: {} 2024-04-09T21:59:59.962 DEBUG:boto:Token: None 2024-04-09T21:59:59.962 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.963 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aMTyHohDjfz8o7IqTWZ1eiEclrQ= 2024-04-09T21:59:59.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aMTyHohDjfz8o7IqTWZ1eiEclrQ=', 'Content-Length': '0'} 2024-04-09T21:59:59.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009f50f47afa840b83-006615ba5f-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:59.965 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.965 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.965 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.965 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.965 DEBUG:boto:Method: GET 2024-04-09T21:59:59.965 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.965 DEBUG:boto:Data: 2024-04-09T21:59:59.965 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.965 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.965 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.966 DEBUG:boto:Params: {} 2024-04-09T21:59:59.966 DEBUG:boto:Token: None 2024-04-09T21:59:59.966 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.966 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8= 2024-04-09T21:59:59.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8=', 'Content-Length': '0'} 2024-04-09T21:59:59.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a5375ac815774c78-006615ba5f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.968 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.968 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.968 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.968 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.968 DEBUG:boto:Method: GET 2024-04-09T21:59:59.968 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.968 DEBUG:boto:Data: 2024-04-09T21:59:59.968 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.968 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.969 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.969 DEBUG:boto:Params: {} 2024-04-09T21:59:59.969 DEBUG:boto:Token: None 2024-04-09T21:59:59.969 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.969 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8= 2024-04-09T21:59:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8=', 'Content-Length': '0'} 2024-04-09T21:59:59.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000369f08b8ab6febcd-006615ba5f-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.971 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.971 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.971 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.971 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.972 DEBUG:boto:Method: GET 2024-04-09T21:59:59.972 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.972 DEBUG:boto:Data: 2024-04-09T21:59:59.972 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.972 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.972 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.972 DEBUG:boto:Params: {} 2024-04-09T21:59:59.972 DEBUG:boto:Token: None 2024-04-09T21:59:59.972 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/ 2024-04-09T21:59:59.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X6N9PA76GjjDCxjSkmYxhoh0Tgo= 2024-04-09T21:59:59.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X6N9PA76GjjDCxjSkmYxhoh0Tgo=', 'Content-Length': '0'} 2024-04-09T21:59:59.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9e9114101371244-006615ba5f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.975 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:59.976 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.976 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.976 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.976 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.976 DEBUG:boto:Method: GET 2024-04-09T21:59:59.976 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.976 DEBUG:boto:Data: 2024-04-09T21:59:59.976 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.976 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.976 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.976 DEBUG:boto:Params: {} 2024-04-09T21:59:59.977 DEBUG:boto:Token: None 2024-04-09T21:59:59.977 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/ 2024-04-09T21:59:59.977 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X6N9PA76GjjDCxjSkmYxhoh0Tgo= 2024-04-09T21:59:59.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X6N9PA76GjjDCxjSkmYxhoh0Tgo=', 'Content-Length': '0'} 2024-04-09T21:59:59.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e42476de9b61f3f7-006615ba5f-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.979 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T21:59:59.979 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.979 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.979 DEBUG:boto:Method: HEAD 2024-04-09T21:59:59.980 DEBUG:boto:Path: /xryqqk-15/dummy 2024-04-09T21:59:59.980 DEBUG:boto:Data: 2024-04-09T21:59:59.980 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.980 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.980 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.980 DEBUG:boto:Params: {} 2024-04-09T21:59:59.980 DEBUG:boto:Token: None 2024-04-09T21:59:59.980 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy 2024-04-09T21:59:59.980 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ysTEmxDblpHI+NIhfUv5TZe2DdM= 2024-04-09T21:59:59.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ysTEmxDblpHI+NIhfUv5TZe2DdM=', 'Content-Length': '0'} 2024-04-09T21:59:59.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002bc5402616d8ad2d-006615ba5f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.983 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.983 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.983 DEBUG:boto:Method: HEAD 2024-04-09T21:59:59.983 DEBUG:boto:Path: /xryqqk-15/dummy 2024-04-09T21:59:59.983 DEBUG:boto:Data: 2024-04-09T21:59:59.983 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.983 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.983 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.983 DEBUG:boto:Params: {} 2024-04-09T21:59:59.983 DEBUG:boto:Token: None 2024-04-09T21:59:59.984 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy 2024-04-09T21:59:59.984 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ysTEmxDblpHI+NIhfUv5TZe2DdM= 2024-04-09T21:59:59.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ysTEmxDblpHI+NIhfUv5TZe2DdM=', 'Content-Length': '0'} 2024-04-09T21:59:59.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048eaf80750d89869-006615ba5f-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T21:59:59.986 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.986 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.986 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.987 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.987 DEBUG:boto:Method: GET 2024-04-09T21:59:59.987 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.987 DEBUG:boto:Data: 2024-04-09T21:59:59.987 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.987 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.987 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.987 DEBUG:boto:Params: {} 2024-04-09T21:59:59.987 DEBUG:boto:Token: None 2024-04-09T21:59:59.987 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.987 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8= 2024-04-09T21:59:59.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8=', 'Content-Length': '0'} 2024-04-09T21:59:59.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c73bcaef302766b-006615ba5f-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.989 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T21:59:59.989 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T21:59:59.989 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.989 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.990 DEBUG:boto:Method: GET 2024-04-09T21:59:59.990 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.990 DEBUG:boto:Data: 2024-04-09T21:59:59.990 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.990 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T21:59:59.990 DEBUG:boto:Port: 8000 2024-04-09T21:59:59.990 DEBUG:boto:Params: {} 2024-04-09T21:59:59.990 DEBUG:boto:Token: None 2024-04-09T21:59:59.990 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T21:59:59.990 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8= 2024-04-09T21:59:59.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2fhjZa43uT/+ohaoFDkOfsiZ5s8=', 'Content-Length': '0'} 2024-04-09T21:59:59.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039298b7c4863c56b-006615ba5f-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T21:59:59.992 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T21:59:59.992 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T21:59:59.992 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.992 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.993 DEBUG:boto:Method: GET 2024-04-09T21:59:59.993 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T21:59:59.993 DEBUG:boto:Data: 2024-04-09T21:59:59.993 DEBUG:boto:Headers: {} 2024-04-09T21:59:59.993 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T21:59:59.993 DEBUG:boto:Port: 8001 2024-04-09T21:59:59.993 DEBUG:boto:Params: {} 2024-04-09T21:59:59.993 DEBUG:boto:Token: None 2024-04-09T21:59:59.993 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 21:59:59 GMT /xryqqk-15/ 2024-04-09T21:59:59.993 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:X6N9PA76GjjDCxjSkmYxhoh0Tgo= 2024-04-09T21:59:59.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:59:59 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X6N9PA76GjjDCxjSkmYxhoh0Tgo=', 'Content-Length': '0'} 2024-04-09T21:59:59.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e05fd203e47061-006615ba5f-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:59:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.042 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:00:00.043 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.043 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.043 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T22:00:00.043 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T22:00:00.043 DEBUG:boto:Method: GET 2024-04-09T22:00:00.043 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T22:00:00.043 DEBUG:boto:Data: 2024-04-09T22:00:00.043 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.044 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.044 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.044 DEBUG:boto:Params: {} 2024-04-09T22:00:00.044 DEBUG:boto:Token: None 2024-04-09T22:00:00.044 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cXBVV8xPb96umAGAbwko+XkSoIE= 2024-04-09T22:00:00.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cXBVV8xPb96umAGAbwko+XkSoIE=', 'Content-Length': '0'} 2024-04-09T22:00:00.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c0c733b7a9226b6-006615ba60-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.047 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:00:00.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-15 zones={a2, a1} 2024-04-09T22:00:00.047 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.047 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.047 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.047 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.047 DEBUG:boto:Data: 2024-04-09T22:00:00.048 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.048 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.048 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.048 DEBUG:boto:Params: {} 2024-04-09T22:00:00.048 DEBUG:boto:Token: None 2024-04-09T22:00:00.048 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.048 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061e8b3b6e13c02ec-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.051 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.051 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.051 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.051 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.051 DEBUG:boto:Data: 2024-04-09T22:00:00.051 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.051 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.051 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.051 DEBUG:boto:Params: {} 2024-04-09T22:00:00.051 DEBUG:boto:Token: None 2024-04-09T22:00:00.052 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.052 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-amz-request-id', 'tx0000030b5ae7e8ccfe153-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.063 DEBUG:rgw_multi.tests:zone=a1 version=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:00:00.063 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.064 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.064 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.064 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.064 DEBUG:boto:Data: 2024-04-09T22:00:00.064 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.064 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.064 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.064 DEBUG:boto:Params: {} 2024-04-09T22:00:00.064 DEBUG:boto:Token: None 2024-04-09T22:00:00.065 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.065 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.067 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099e0fc6b1c3662f5-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.067 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.067 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.067 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.068 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.068 DEBUG:boto:Data: 2024-04-09T22:00:00.068 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.068 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.068 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.068 DEBUG:boto:Params: {} 2024-04-09T22:00:00.068 DEBUG:boto:Token: None 2024-04-09T22:00:00.068 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.068 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-amz-request-id', 'tx00000282921702800a6dd-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.079 DEBUG:rgw_multi.tests:zone=a2 version=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:00:00.079 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.079 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.079 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.079 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.079 DEBUG:boto:Data: 2024-04-09T22:00:00.079 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.080 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.080 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.080 DEBUG:boto:Params: {} 2024-04-09T22:00:00.080 DEBUG:boto:Token: None 2024-04-09T22:00:00.080 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.080 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fb43d4ac5776a70-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.083 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.083 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.083 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.083 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.083 DEBUG:boto:Data: 2024-04-09T22:00:00.083 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.083 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.083 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.083 DEBUG:boto:Params: {} 2024-04-09T22:00:00.083 DEBUG:boto:Token: None 2024-04-09T22:00:00.084 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.084 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-amz-request-id', 'tx00000f7ccfd9d070c76b2-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.131 DEBUG:rgw_multi.tests:zone=a1 version=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:00:00.131 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.131 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.131 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.131 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.131 DEBUG:boto:Data: 2024-04-09T22:00:00.131 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.131 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.131 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.131 DEBUG:boto:Params: {} 2024-04-09T22:00:00.131 DEBUG:boto:Token: None 2024-04-09T22:00:00.132 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.132 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.134 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000426f72f53c0be48f-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.134 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.134 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.135 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.135 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.135 DEBUG:boto:Data: 2024-04-09T22:00:00.135 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.135 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.135 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.135 DEBUG:boto:Params: {} 2024-04-09T22:00:00.135 DEBUG:boto:Token: None 2024-04-09T22:00:00.135 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.135 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-amz-request-id', 'tx000009217087e04034fd0-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.147 DEBUG:rgw_multi.tests:zone=a2 version=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:00:00.147 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.147 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.147 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.147 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.147 DEBUG:boto:Data: 2024-04-09T22:00:00.148 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.148 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.148 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.148 DEBUG:boto:Params: {} 2024-04-09T22:00:00.148 DEBUG:boto:Token: None 2024-04-09T22:00:00.148 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.148 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.150 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8768fef1df04632-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.151 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.151 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.151 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.151 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.151 DEBUG:boto:Data: 2024-04-09T22:00:00.151 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.151 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.151 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.152 DEBUG:boto:Params: {} 2024-04-09T22:00:00.152 DEBUG:boto:Token: None 2024-04-09T22:00:00.152 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.152 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-amz-request-id', 'tx000007789d51195e14e6a-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.161 DEBUG:rgw_multi.tests:zone=a1 version=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:00:00.161 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.161 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.161 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.161 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.161 DEBUG:boto:Data: 2024-04-09T22:00:00.162 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.162 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.162 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.162 DEBUG:boto:Params: {} 2024-04-09T22:00:00.162 DEBUG:boto:Token: None 2024-04-09T22:00:00.162 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.162 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c392cdc4a7ce05f-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.164 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.164 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.165 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.165 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.165 DEBUG:boto:Data: 2024-04-09T22:00:00.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.165 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.165 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.165 DEBUG:boto:Params: {} 2024-04-09T22:00:00.165 DEBUG:boto:Token: None 2024-04-09T22:00:00.165 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.165 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.211 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-amz-request-id', 'tx0000013fa7e9cf5334686-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.211 DEBUG:rgw_multi.tests:zone=a2 version=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:00:00.211 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.211 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.211 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.211 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.211 DEBUG:boto:Data: 2024-04-09T22:00:00.211 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.211 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.211 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.212 DEBUG:boto:Params: {} 2024-04-09T22:00:00.212 DEBUG:boto:Token: None 2024-04-09T22:00:00.212 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.212 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfb0c480fdaa35d0-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.214 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.214 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.214 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.215 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.215 DEBUG:boto:Data: 2024-04-09T22:00:00.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.215 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.215 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.215 DEBUG:boto:Params: {} 2024-04-09T22:00:00.215 DEBUG:boto:Token: None 2024-04-09T22:00:00.215 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.215 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-amz-request-id', 'tx0000036cd56566a89f403-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.223 DEBUG:rgw_multi.tests:zone=a1 version=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:00:00.224 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.224 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.224 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.224 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.224 DEBUG:boto:Data: 2024-04-09T22:00:00.224 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.224 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.224 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.224 DEBUG:boto:Params: {} 2024-04-09T22:00:00.224 DEBUG:boto:Token: None 2024-04-09T22:00:00.225 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.225 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7a301567ed176a6-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.227 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.227 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.227 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.227 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.227 DEBUG:boto:Data: 2024-04-09T22:00:00.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.228 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.228 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.228 DEBUG:boto:Params: {} 2024-04-09T22:00:00.228 DEBUG:boto:Token: None 2024-04-09T22:00:00.228 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.228 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-amz-request-id', 'tx000004b9b4697d0034270-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.239 DEBUG:rgw_multi.tests:zone=a2 version=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:00:00.239 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.239 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.239 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.239 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.239 DEBUG:boto:Data: 2024-04-09T22:00:00.239 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.239 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.240 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.240 DEBUG:boto:Params: {} 2024-04-09T22:00:00.240 DEBUG:boto:Token: None 2024-04-09T22:00:00.240 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000562244436ed7a4f5-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.246 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.246 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.247 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.247 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.247 DEBUG:boto:Data: 2024-04-09T22:00:00.247 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.247 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.247 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.247 DEBUG:boto:Params: {} 2024-04-09T22:00:00.247 DEBUG:boto:Token: None 2024-04-09T22:00:00.247 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.248 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-amz-request-id', 'tx00000764d03b01b27e276-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.291 DEBUG:rgw_multi.tests:zone=a1 version=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:00:00.291 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.291 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.291 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.292 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.292 DEBUG:boto:Data: 2024-04-09T22:00:00.292 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.292 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.292 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.292 DEBUG:boto:Params: {} 2024-04-09T22:00:00.292 DEBUG:boto:Token: None 2024-04-09T22:00:00.292 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.292 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025aef3496b1199b2-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.295 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.295 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.295 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.295 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.295 DEBUG:boto:Data: 2024-04-09T22:00:00.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.296 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.296 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.296 DEBUG:boto:Params: {} 2024-04-09T22:00:00.296 DEBUG:boto:Token: None 2024-04-09T22:00:00.296 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.297 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-amz-request-id', 'tx00000ab09473cb025499d-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.339 DEBUG:rgw_multi.tests:zone=a2 version=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:00:00.339 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.339 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.339 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.339 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.339 DEBUG:boto:Data: 2024-04-09T22:00:00.340 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.340 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.340 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.340 DEBUG:boto:Params: {} 2024-04-09T22:00:00.340 DEBUG:boto:Token: None 2024-04-09T22:00:00.340 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.340 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c9f0eb84a831a40-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.343 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.343 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.343 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.343 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.343 DEBUG:boto:Data: 2024-04-09T22:00:00.343 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.343 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.343 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.343 DEBUG:boto:Params: {} 2024-04-09T22:00:00.343 DEBUG:boto:Token: None 2024-04-09T22:00:00.344 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.344 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-amz-request-id', 'tx0000081c58b55996957de-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.353 DEBUG:rgw_multi.tests:zone=a1 version=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:00:00.353 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.353 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.353 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.354 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.354 DEBUG:boto:Data: 2024-04-09T22:00:00.354 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.354 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.354 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.354 DEBUG:boto:Params: {} 2024-04-09T22:00:00.354 DEBUG:boto:Token: None 2024-04-09T22:00:00.354 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.354 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd38d75308a12a65-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.357 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.357 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.357 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.357 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.357 DEBUG:boto:Data: 2024-04-09T22:00:00.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.357 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.357 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.357 DEBUG:boto:Params: {} 2024-04-09T22:00:00.357 DEBUG:boto:Token: None 2024-04-09T22:00:00.358 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.358 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-amz-request-id', 'tx00000f317aa9b6128aeef-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.365 DEBUG:rgw_multi.tests:zone=a2 version=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:00:00.365 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.366 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.366 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.366 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.366 DEBUG:boto:Data: 2024-04-09T22:00:00.366 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.366 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.366 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.366 DEBUG:boto:Params: {} 2024-04-09T22:00:00.366 DEBUG:boto:Token: None 2024-04-09T22:00:00.367 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.367 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ece047333d579ab-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.369 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.369 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.370 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.370 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.370 DEBUG:boto:Data: 2024-04-09T22:00:00.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.370 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.370 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.370 DEBUG:boto:Params: {} 2024-04-09T22:00:00.370 DEBUG:boto:Token: None 2024-04-09T22:00:00.370 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.371 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-amz-request-id', 'tx000003f44de010783b521-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.379 DEBUG:rgw_multi.tests:zone=a1 version=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:00:00.379 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.379 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.379 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.379 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.379 DEBUG:boto:Data: 2024-04-09T22:00:00.379 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.380 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.380 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.380 DEBUG:boto:Params: {} 2024-04-09T22:00:00.380 DEBUG:boto:Token: None 2024-04-09T22:00:00.380 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.380 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069b985e5ff271b2c-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.382 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.383 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.383 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.383 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.383 DEBUG:boto:Data: 2024-04-09T22:00:00.383 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.383 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.383 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.383 DEBUG:boto:Params: {} 2024-04-09T22:00:00.383 DEBUG:boto:Token: None 2024-04-09T22:00:00.383 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.383 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.426 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-amz-request-id', 'tx00000895a46e46661c3fb-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.427 DEBUG:rgw_multi.tests:zone=a2 version=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:00:00.427 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.427 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.427 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.427 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.427 DEBUG:boto:Data: 2024-04-09T22:00:00.427 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.427 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.427 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.427 DEBUG:boto:Params: {} 2024-04-09T22:00:00.428 DEBUG:boto:Token: None 2024-04-09T22:00:00.428 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.428 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038292f884f3b53dc-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.430 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.430 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.430 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.430 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.430 DEBUG:boto:Data: 2024-04-09T22:00:00.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.431 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.431 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.431 DEBUG:boto:Params: {} 2024-04-09T22:00:00.431 DEBUG:boto:Token: None 2024-04-09T22:00:00.431 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.431 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-amz-request-id', 'tx000008cfefbf03ca2ff8a-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.441 DEBUG:rgw_multi.tests:zone=a1 version=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:00:00.441 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.441 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.441 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.441 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.441 DEBUG:boto:Data: 2024-04-09T22:00:00.441 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.442 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.442 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.442 DEBUG:boto:Params: {} 2024-04-09T22:00:00.442 DEBUG:boto:Token: None 2024-04-09T22:00:00.442 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.442 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f244bc36b43215e4-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.444 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.444 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.445 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.445 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.445 DEBUG:boto:Data: 2024-04-09T22:00:00.445 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.445 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.445 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.445 DEBUG:boto:Params: {} 2024-04-09T22:00:00.445 DEBUG:boto:Token: None 2024-04-09T22:00:00.445 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.446 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.455 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-amz-request-id', 'tx0000065379a96b62f6311-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.455 DEBUG:rgw_multi.tests:zone=a2 version=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:00:00.456 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.456 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.456 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.456 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.456 DEBUG:boto:Data: 2024-04-09T22:00:00.456 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.456 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.456 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.456 DEBUG:boto:Params: {} 2024-04-09T22:00:00.456 DEBUG:boto:Token: None 2024-04-09T22:00:00.456 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.457 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225d4da4adb44598-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.459 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.459 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.459 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.459 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.459 DEBUG:boto:Data: 2024-04-09T22:00:00.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.459 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.460 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.460 DEBUG:boto:Params: {} 2024-04-09T22:00:00.460 DEBUG:boto:Token: None 2024-04-09T22:00:00.460 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-amz-request-id', 'tx00000e5b85be73e55838a-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.503 DEBUG:rgw_multi.tests:zone=a1 version=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:00:00.503 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.503 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.503 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.503 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.503 DEBUG:boto:Data: 2024-04-09T22:00:00.503 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.503 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.503 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.503 DEBUG:boto:Params: {} 2024-04-09T22:00:00.504 DEBUG:boto:Token: None 2024-04-09T22:00:00.504 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.504 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001be90b1fd20ef587-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.506 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.506 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.507 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.507 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.507 DEBUG:boto:Data: 2024-04-09T22:00:00.507 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.507 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.507 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.507 DEBUG:boto:Params: {} 2024-04-09T22:00:00.507 DEBUG:boto:Token: None 2024-04-09T22:00:00.507 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.507 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-amz-request-id', 'tx00000e6abffe731aeec84-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.516 DEBUG:rgw_multi.tests:zone=a2 version=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:00:00.516 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.516 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.516 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.516 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.516 DEBUG:boto:Data: 2024-04-09T22:00:00.517 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.517 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.517 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.517 DEBUG:boto:Params: {} 2024-04-09T22:00:00.517 DEBUG:boto:Token: None 2024-04-09T22:00:00.517 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.517 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec94a36ff29d532e-006615ba60-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.519 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.520 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.520 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.520 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.520 DEBUG:boto:Data: 2024-04-09T22:00:00.520 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.520 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:00:00.520 DEBUG:boto:Port: 8000 2024-04-09T22:00:00.520 DEBUG:boto:Params: {} 2024-04-09T22:00:00.520 DEBUG:boto:Token: None 2024-04-09T22:00:00.520 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.520 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-amz-request-id', 'tx00000d9a027b25c9f40d0-006615ba60-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.529 DEBUG:rgw_multi.tests:zone=a1 version=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:00:00.529 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:00:00.529 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:00:00.529 DEBUG:boto:Method: HEAD 2024-04-09T22:00:00.529 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:00:00.529 DEBUG:boto:Data: 2024-04-09T22:00:00.529 DEBUG:boto:Headers: {} 2024-04-09T22:00:00.529 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.530 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.530 DEBUG:boto:Params: {} 2024-04-09T22:00:00.530 DEBUG:boto:Token: None 2024-04-09T22:00:00.530 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/ 2024-04-09T22:00:00.530 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s= 2024-04-09T22:00:00.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ohqMdSL1ynnxtyPyqxqFtetsx/s=', 'Content-Length': '0'} 2024-04-09T22:00:00.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d562b997b385d94e-006615ba60-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.532 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:00:00.532 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:00:00.533 DEBUG:boto:Method: PUT 2024-04-09T22:00:00.533 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:00:00.533 DEBUG:boto:Data: 2024-04-09T22:00:00.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-09T22:00:00.533 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:00:00.533 DEBUG:boto:Port: 8001 2024-04-09T22:00:00.533 DEBUG:boto:Params: {} 2024-04-09T22:00:00.533 DEBUG:boto:Token: None 2024-04-09T22:00:00.533 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Tue, 09 Apr 2024 22:00:00 GMT /xryqqk-15/obj 2024-04-09T22:00:00.533 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI= 2024-04-09T22:00:00.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:00:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OwP7qZzStqAg4VNDGPAFcKA5+MI='} 2024-04-09T22:00:00.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-amz-request-id', 'tx00000e47451408f373a9d-006615ba60-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:00:00.542 DEBUG:rgw_multi.tests:zone=a2 version=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:00:00.542 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-15 2024-04-09T22:00:00.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:00:00.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:00:00.543 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:00:00.599 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:00:00.599 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:00:00.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:00.633+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:00.634 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2024-04-09T22:00:00.634 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:05.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:05.634+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:05.635 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2024-04-09T22:00:05.636 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:10.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:10.635+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:10.637 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2024-04-09T22:00:10.637 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:15.637+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:15.638 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , , , ] 2024-04-09T22:00:15.638 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:20.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:20.638+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:20.639 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000009.41.6, , ] 2024-04-09T22:00:20.640 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:25.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:25.639+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:25.641 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000009.41.6, , ] 2024-04-09T22:00:25.641 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:30.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:30.640+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:30.642 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000009.41.6, , ] 2024-04-09T22:00:30.642 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:35.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:35.642+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:35.647 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000009.41.6, , ] 2024-04-09T22:00:35.647 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:40.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:40.643+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:40.645 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000013.49.6, , ] 2024-04-09T22:00:40.645 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:45.644+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:45.646 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000013.49.6, , ] 2024-04-09T22:00:45.646 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:50.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:50.646+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:50.647 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000013.49.6, , ] 2024-04-09T22:00:50.647 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:00:55.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:00:55.647+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:00:55.648 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000013.49.6, , ] 2024-04-09T22:00:55.648 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:00.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:00.648+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:00.650 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:00.650 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:05.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:05.649+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:05.651 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:05.651 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:10.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:10.651+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:10.652 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:10.652 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:15.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:15.652+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:15.654 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:15.654 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:20.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:20.653+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:20.655 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:20.655 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:25.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:25.654+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:25.656 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:25.656 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:30.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:30.656+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:30.657 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:30.657 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:35.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:35.657+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:35.658 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:35.658 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:40.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:40.658+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:40.659 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:40.659 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:45.659+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:45.660 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:45.660 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:50.660+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:50.662 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:50.662 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:01:55.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:01:55.662+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:01:55.663 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:01:55.663 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:02:00.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:00.663+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:00.664 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:02:00.664 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:02:05.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:05.664+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:05.666 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:02:05.666 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:02:10.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:10.665+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:10.668 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:02:10.668 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:02:15.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:15.667+0000 7fca9c417780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:15.668 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000017.57.6, , ] 2024-04-09T22:02:15.668 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:02:20.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:20.668+0000 7fca9c417780 1 bucket sync caught up with source: 2024-04-09T22:02:20.670 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000003.9.10, , , 00000000031.93.9, , ] 2024-04-09T22:02:20.670 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000003.9.10, , , 00000000020.62.12, , ] 2024-04-09T22:02:20.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:20.668+0000 7fca9c417780 0 bucket checkpoint complete 2024-04-09T22:02:20.686 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-15 2024-04-09T22:02:20.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:02:20.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:02:20.687 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:02:20.751 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:20.752 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:20.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:20.791+0000 7f74d5fc4780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:20.793 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.11.11, , , 00000000028.99.6, , ] 2024-04-09T22:02:20.793 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.11.11, , , 00000000031.106.12, , ] 2024-04-09T22:02:25.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:25.792+0000 7f74d5fc4780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:25.794 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.11.11, , , 00000000028.99.6, , ] 2024-04-09T22:02:25.794 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.11.11, , , 00000000031.106.12, , ] 2024-04-09T22:02:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:30.794+0000 7f74d5fc4780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:30.795 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.11.11, , , 00000000028.99.6, , ] 2024-04-09T22:02:30.795 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.11.11, , , 00000000031.106.12, , ] 2024-04-09T22:02:35.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:35.795+0000 7f74d5fc4780 1 waiting for incremental sync to catch up: 2024-04-09T22:02:35.797 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.11.11, , , 00000000028.99.6, , ] 2024-04-09T22:02:35.797 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.11.11, , , 00000000031.106.12, , ] 2024-04-09T22:02:40.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:40.797+0000 7f74d5fc4780 1 bucket sync caught up with source: 2024-04-09T22:02:40.798 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , 00000000002.11.11, , , 00000000031.106.12, , ] 2024-04-09T22:02:40.798 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , 00000000002.11.11, , , 00000000031.106.12, , ] 2024-04-09T22:02:40.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:40.797+0000 7f74d5fc4780 0 bucket checkpoint complete 2024-04-09T22:02:40.814 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-15 zones={a2, a1} 2024-04-09T22:02:40.815 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.815 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.815 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.815 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:02:40.815 DEBUG:boto:Data: 2024-04-09T22:02:40.815 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.815 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.816 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.816 DEBUG:boto:Params: {} 2024-04-09T22:02:40.816 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:02:40.816 DEBUG:boto:Token: None 2024-04-09T22:02:40.817 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.817 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1+K9RxJL84ucVW2lLN9j7/3Ocvk= 2024-04-09T22:02:40.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1+K9RxJL84ucVW2lLN9j7/3Ocvk=', 'Content-Length': '0'} 2024-04-09T22:02:40.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bc5f0f0b18304be-006615bb00-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.822 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.822 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.822 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.822 DEBUG:boto:Path: /xryqqk-15/ 2024-04-09T22:02:40.822 DEBUG:boto:Data: 2024-04-09T22:02:40.822 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.822 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.822 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.822 DEBUG:boto:Params: {} 2024-04-09T22:02:40.823 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:02:40.823 DEBUG:boto:Token: None 2024-04-09T22:02:40.823 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.823 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1+K9RxJL84ucVW2lLN9j7/3Ocvk= 2024-04-09T22:02:40.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1+K9RxJL84ucVW2lLN9j7/3Ocvk=', 'Content-Length': '0'} 2024-04-09T22:02:40.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000825434d3c361d0dc-006615bb00-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.827 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:02:40.827 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.827 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.827 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.827 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.828 DEBUG:boto:Method: GET 2024-04-09T22:02:40.828 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.828 DEBUG:boto:Data: 2024-04-09T22:02:40.828 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.828 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.828 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.828 DEBUG:boto:Params: {} 2024-04-09T22:02:40.828 DEBUG:boto:Token: None 2024-04-09T22:02:40.828 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/?versions 2024-04-09T22:02:40.828 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM= 2024-04-09T22:02:40.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM=', 'Content-Length': '0'} 2024-04-09T22:02:40.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000940d12f2fc7c36a5-006615bb00-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.831 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKrgu-QkcVxiHciJpP8vdr7GSowJPRVUtrue2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWlVOuOCaDVTTkuos9UkExpZN5xD4yMofalse2024-04-09T22:00:00.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJduizZozTy1Fs0rff.ccLhHDDTufRNyfalse2024-04-09T22:00:00.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTBks51TGviAJ2FKkudDkJCF6TK7A.M1false2024-04-09T22:00:00.509Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9i4vn5lmuYHa8oi6UyZDWS8fqdzaphqfalse2024-04-09T22:00:00.447Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9iw.TDO7FhgnqwpVBEInTJKvjjyNOlWfalse2024-04-09T22:00:00.433Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje0tEUBVITt61ncOaHmD8QxbssFQjO92false2024-04-09T22:00:00.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqS-jaSb9pCWrKjwAQ6neHckrgYDdBl3false2024-04-09T22:00:00.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ3pYxHY6NfaDDMPsw0PjqBhuWJT5n5Efalse2024-04-09T22:00:00.345Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnR0R8ccnFLtMtmKXAhGHxsgDs8klFV8false2024-04-09T22:00:00.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9.828cYFy0Yv6yg.oVxTKG-.bzjWzynfalse2024-04-09T22:00:00.298Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNGL-GxJyNdNy67KIPsaQVKx4ZMNZGZDfalse2024-04-09T22:00:00.250Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.rWxN.f1EH02C.PAOjorQnSaCM.GZJPfalse2024-04-09T22:00:00.217Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbIbsr3kXomqanFnhcQh4V9hg52sTgxnfalse2024-04-09T22:00:00.230Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3szGQVHc-46oVFHsOBsQWeWjtcNxuwLfalse2024-04-09T22:00:00.167Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQCLRNCVITG24MAEWwoGf.BsQb23bjCefalse2024-04-09T22:00:00.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qxfalse2024-04-09T22:00:00.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.sV.ddadxTWP-UmBEgRWXYXeri-9vofalse2024-04-09T22:00:00.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuHIeCAUenOwzrzlgUIRrkkwqPwkwVvcfalse2024-04-09T22:00:00.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzQzhh-6iBfnwpGd2KJmyy3OGGGRyItLfalse2024-04-09T22:00:00.070Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.833 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T22:02:40.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.833 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.835 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:02:40.836 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.836 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.836 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.836 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.836 DEBUG:boto:Method: GET 2024-04-09T22:02:40.836 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.836 DEBUG:boto:Data: 2024-04-09T22:02:40.836 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.836 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.836 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.836 DEBUG:boto:Params: {} 2024-04-09T22:02:40.837 DEBUG:boto:Token: None 2024-04-09T22:02:40.837 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/?versions 2024-04-09T22:02:40.837 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM= 2024-04-09T22:02:40.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM=', 'Content-Length': '0'} 2024-04-09T22:02:40.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c02f41f83c6e4f4e-006615bb00-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.840 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKrgu-QkcVxiHciJpP8vdr7GSowJPRVUtrue2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWlVOuOCaDVTTkuos9UkExpZN5xD4yMofalse2024-04-09T22:00:00.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJduizZozTy1Fs0rff.ccLhHDDTufRNyfalse2024-04-09T22:00:00.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTBks51TGviAJ2FKkudDkJCF6TK7A.M1false2024-04-09T22:00:00.509Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9i4vn5lmuYHa8oi6UyZDWS8fqdzaphqfalse2024-04-09T22:00:00.447Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9iw.TDO7FhgnqwpVBEInTJKvjjyNOlWfalse2024-04-09T22:00:00.433Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje0tEUBVITt61ncOaHmD8QxbssFQjO92false2024-04-09T22:00:00.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqS-jaSb9pCWrKjwAQ6neHckrgYDdBl3false2024-04-09T22:00:00.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ3pYxHY6NfaDDMPsw0PjqBhuWJT5n5Efalse2024-04-09T22:00:00.345Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnR0R8ccnFLtMtmKXAhGHxsgDs8klFV8false2024-04-09T22:00:00.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9.828cYFy0Yv6yg.oVxTKG-.bzjWzynfalse2024-04-09T22:00:00.298Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNGL-GxJyNdNy67KIPsaQVKx4ZMNZGZDfalse2024-04-09T22:00:00.250Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.rWxN.f1EH02C.PAOjorQnSaCM.GZJPfalse2024-04-09T22:00:00.217Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbIbsr3kXomqanFnhcQh4V9hg52sTgxnfalse2024-04-09T22:00:00.230Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3szGQVHc-46oVFHsOBsQWeWjtcNxuwLfalse2024-04-09T22:00:00.167Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQCLRNCVITG24MAEWwoGf.BsQb23bjCefalse2024-04-09T22:00:00.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qxfalse2024-04-09T22:00:00.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.sV.ddadxTWP-UmBEgRWXYXeri-9vofalse2024-04-09T22:00:00.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuHIeCAUenOwzrzlgUIRrkkwqPwkwVvcfalse2024-04-09T22:00:00.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzQzhh-6iBfnwpGd2KJmyy3OGGGRyItLfalse2024-04-09T22:00:00.070Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:02:40.842 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.843 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.843 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.843 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.843 DEBUG:boto:Method: GET 2024-04-09T22:02:40.843 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.843 DEBUG:boto:Data: 2024-04-09T22:02:40.843 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.843 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.843 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.843 DEBUG:boto:Params: {} 2024-04-09T22:02:40.843 DEBUG:boto:Token: None 2024-04-09T22:02:40.844 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/?versions 2024-04-09T22:02:40.844 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM= 2024-04-09T22:02:40.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM=', 'Content-Length': '0'} 2024-04-09T22:02:40.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a39fd08413e8872e-006615bb00-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.846 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKrgu-QkcVxiHciJpP8vdr7GSowJPRVUtrue2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWlVOuOCaDVTTkuos9UkExpZN5xD4yMofalse2024-04-09T22:00:00.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJduizZozTy1Fs0rff.ccLhHDDTufRNyfalse2024-04-09T22:00:00.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTBks51TGviAJ2FKkudDkJCF6TK7A.M1false2024-04-09T22:00:00.509Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9i4vn5lmuYHa8oi6UyZDWS8fqdzaphqfalse2024-04-09T22:00:00.447Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9iw.TDO7FhgnqwpVBEInTJKvjjyNOlWfalse2024-04-09T22:00:00.433Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje0tEUBVITt61ncOaHmD8QxbssFQjO92false2024-04-09T22:00:00.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqS-jaSb9pCWrKjwAQ6neHckrgYDdBl3false2024-04-09T22:00:00.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ3pYxHY6NfaDDMPsw0PjqBhuWJT5n5Efalse2024-04-09T22:00:00.345Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnR0R8ccnFLtMtmKXAhGHxsgDs8klFV8false2024-04-09T22:00:00.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9.828cYFy0Yv6yg.oVxTKG-.bzjWzynfalse2024-04-09T22:00:00.298Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNGL-GxJyNdNy67KIPsaQVKx4ZMNZGZDfalse2024-04-09T22:00:00.250Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.rWxN.f1EH02C.PAOjorQnSaCM.GZJPfalse2024-04-09T22:00:00.217Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbIbsr3kXomqanFnhcQh4V9hg52sTgxnfalse2024-04-09T22:00:00.230Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3szGQVHc-46oVFHsOBsQWeWjtcNxuwLfalse2024-04-09T22:00:00.167Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQCLRNCVITG24MAEWwoGf.BsQb23bjCefalse2024-04-09T22:00:00.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qxfalse2024-04-09T22:00:00.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.sV.ddadxTWP-UmBEgRWXYXeri-9vofalse2024-04-09T22:00:00.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuHIeCAUenOwzrzlgUIRrkkwqPwkwVvcfalse2024-04-09T22:00:00.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzQzhh-6iBfnwpGd2KJmyy3OGGGRyItLfalse2024-04-09T22:00:00.070Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.847 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.848 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.848 DEBUG:boto:path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.848 DEBUG:boto:auth_path=/xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.848 DEBUG:boto:Method: GET 2024-04-09T22:02:40.848 DEBUG:boto:Path: /xryqqk-15/?versions&max-keys=999 2024-04-09T22:02:40.848 DEBUG:boto:Data: 2024-04-09T22:02:40.848 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.848 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.848 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.848 DEBUG:boto:Params: {} 2024-04-09T22:02:40.848 DEBUG:boto:Token: None 2024-04-09T22:02:40.849 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/?versions 2024-04-09T22:02:40.849 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM= 2024-04-09T22:02:40.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pDSxc4ik2tXJpeKZ5EMVytzgFXM=', 'Content-Length': '0'} 2024-04-09T22:02:40.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea41ee9d7462280-006615bb00-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.851 DEBUG:boto:b'xryqqk-15999falsedummykRroa4vyZJOJjSCIentxh7cKVwENfyGtrue2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjKrgu-QkcVxiHciJpP8vdr7GSowJPRVUtrue2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWlVOuOCaDVTTkuos9UkExpZN5xD4yMofalse2024-04-09T22:00:00.535Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJduizZozTy1Fs0rff.ccLhHDDTufRNyfalse2024-04-09T22:00:00.461Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjTBks51TGviAJ2FKkudDkJCF6TK7A.M1false2024-04-09T22:00:00.509Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9i4vn5lmuYHa8oi6UyZDWS8fqdzaphqfalse2024-04-09T22:00:00.447Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9iw.TDO7FhgnqwpVBEInTJKvjjyNOlWfalse2024-04-09T22:00:00.433Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobje0tEUBVITt61ncOaHmD8QxbssFQjO92false2024-04-09T22:00:00.372Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqS-jaSb9pCWrKjwAQ6neHckrgYDdBl3false2024-04-09T22:00:00.385Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjJ3pYxHY6NfaDDMPsw0PjqBhuWJT5n5Efalse2024-04-09T22:00:00.345Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnR0R8ccnFLtMtmKXAhGHxsgDs8klFV8false2024-04-09T22:00:00.359Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9.828cYFy0Yv6yg.oVxTKG-.bzjWzynfalse2024-04-09T22:00:00.298Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNGL-GxJyNdNy67KIPsaQVKx4ZMNZGZDfalse2024-04-09T22:00:00.250Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj.rWxN.f1EH02C.PAOjorQnSaCM.GZJPfalse2024-04-09T22:00:00.217Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjbIbsr3kXomqanFnhcQh4V9hg52sTgxnfalse2024-04-09T22:00:00.230Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3szGQVHc-46oVFHsOBsQWeWjtcNxuwLfalse2024-04-09T22:00:00.167Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjQCLRNCVITG24MAEWwoGf.BsQb23bjCefalse2024-04-09T22:00:00.154Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qxfalse2024-04-09T22:00:00.137Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ.sV.ddadxTWP-UmBEgRWXYXeri-9vofalse2024-04-09T22:00:00.085Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjuHIeCAUenOwzrzlgUIRrkkwqPwkwVvcfalse2024-04-09T22:00:00.054Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzQzhh-6iBfnwpGd2KJmyy3OGGGRyItLfalse2024-04-09T22:00:00.070Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T22:02:40.853 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.853 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.853 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.853 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.853 DEBUG:boto:Method: GET 2024-04-09T22:02:40.853 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.853 DEBUG:boto:Data: 2024-04-09T22:02:40.853 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.853 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.853 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.854 DEBUG:boto:Params: {} 2024-04-09T22:02:40.854 DEBUG:boto:Token: None 2024-04-09T22:02:40.854 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.854 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw= 2024-04-09T22:02:40.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw=', 'Content-Length': '0'} 2024-04-09T22:02:40.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000053479b50b72eb46f-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.856 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.856 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.856 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.856 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.856 DEBUG:boto:Method: GET 2024-04-09T22:02:40.856 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.856 DEBUG:boto:Data: 2024-04-09T22:02:40.857 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.857 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.857 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.857 DEBUG:boto:Params: {} 2024-04-09T22:02:40.857 DEBUG:boto:Token: None 2024-04-09T22:02:40.857 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.857 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw= 2024-04-09T22:02:40.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw=', 'Content-Length': '0'} 2024-04-09T22:02:40.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b17ed00d333cbcb-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.859 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.859 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.859 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.859 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.860 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.860 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.860 DEBUG:boto:Data: 2024-04-09T22:02:40.860 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.860 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.860 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.860 DEBUG:boto:Params: {} 2024-04-09T22:02:40.860 DEBUG:boto:Token: None 2024-04-09T22:02:40.860 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.860 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bZi120XzflKpRcPP24zjL2uDyx8= 2024-04-09T22:02:40.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bZi120XzflKpRcPP24zjL2uDyx8=', 'Content-Length': '0'} 2024-04-09T22:02:40.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000031166945a859b43c-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.862 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.862 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.862 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.862 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.863 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.863 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.863 DEBUG:boto:Data: 2024-04-09T22:02:40.863 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.863 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.863 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.863 DEBUG:boto:Params: {} 2024-04-09T22:02:40.863 DEBUG:boto:Token: None 2024-04-09T22:02:40.863 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.863 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bZi120XzflKpRcPP24zjL2uDyx8= 2024-04-09T22:02:40.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bZi120XzflKpRcPP24zjL2uDyx8=', 'Content-Length': '0'} 2024-04-09T22:02:40.865 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005b7fd02dda7b1bbf-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T22:02:40.865 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.865 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.865 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.865 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.866 DEBUG:boto:Method: GET 2024-04-09T22:02:40.866 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.866 DEBUG:boto:Data: 2024-04-09T22:02:40.866 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.866 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.866 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.866 DEBUG:boto:Params: {} 2024-04-09T22:02:40.866 DEBUG:boto:Token: None 2024-04-09T22:02:40.866 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.867 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw= 2024-04-09T22:02:40.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw=', 'Content-Length': '0'} 2024-04-09T22:02:40.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006182c1aa18e970f-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.868 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.868 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.869 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.869 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.869 DEBUG:boto:Method: GET 2024-04-09T22:02:40.869 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.869 DEBUG:boto:Data: 2024-04-09T22:02:40.869 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.869 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.869 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.869 DEBUG:boto:Params: {} 2024-04-09T22:02:40.869 DEBUG:boto:Token: None 2024-04-09T22:02:40.870 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.870 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw= 2024-04-09T22:02:40.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw=', 'Content-Length': '0'} 2024-04-09T22:02:40.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000044389317aba0d7dd-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.871 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.872 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.872 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.872 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.872 DEBUG:boto:Method: GET 2024-04-09T22:02:40.872 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.872 DEBUG:boto:Data: 2024-04-09T22:02:40.872 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.872 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.872 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.872 DEBUG:boto:Params: {} 2024-04-09T22:02:40.872 DEBUG:boto:Token: None 2024-04-09T22:02:40.873 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.873 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0= 2024-04-09T22:02:40.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0=', 'Content-Length': '0'} 2024-04-09T22:02:40.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000475643ab43916032-006615bb00-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.875 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.876 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.876 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.876 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.876 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.876 DEBUG:boto:Method: GET 2024-04-09T22:02:40.876 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.876 DEBUG:boto:Data: 2024-04-09T22:02:40.876 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.876 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.877 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.877 DEBUG:boto:Params: {} 2024-04-09T22:02:40.877 DEBUG:boto:Token: None 2024-04-09T22:02:40.877 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.877 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0= 2024-04-09T22:02:40.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0=', 'Content-Length': '0'} 2024-04-09T22:02:40.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d4c615d0de863d3-006615bb00-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.922 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.922 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.923 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.923 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.923 DEBUG:boto:Path: /xryqqk-15/dummy 2024-04-09T22:02:40.923 DEBUG:boto:Data: 2024-04-09T22:02:40.923 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.923 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.923 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.923 DEBUG:boto:Params: {} 2024-04-09T22:02:40.923 DEBUG:boto:Token: None 2024-04-09T22:02:40.923 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy 2024-04-09T22:02:40.924 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hrWVhQc7HFqQGE66rvlJUMCDedA= 2024-04-09T22:02:40.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hrWVhQc7HFqQGE66rvlJUMCDedA=', 'Content-Length': '0'} 2024-04-09T22:02:40.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000effa3a886a4c8d3f-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.926 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.926 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.926 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.926 DEBUG:boto:Path: /xryqqk-15/dummy 2024-04-09T22:02:40.926 DEBUG:boto:Data: 2024-04-09T22:02:40.927 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.927 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.927 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.927 DEBUG:boto:Params: {} 2024-04-09T22:02:40.927 DEBUG:boto:Token: None 2024-04-09T22:02:40.927 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy 2024-04-09T22:02:40.927 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hrWVhQc7HFqQGE66rvlJUMCDedA= 2024-04-09T22:02:40.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hrWVhQc7HFqQGE66rvlJUMCDedA=', 'Content-Length': '0'} 2024-04-09T22:02:40.929 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000987a108909c14a8b-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-09T22:02:40.930 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.930 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.930 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.930 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.930 DEBUG:boto:Method: GET 2024-04-09T22:02:40.930 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.930 DEBUG:boto:Data: 2024-04-09T22:02:40.930 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.930 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.930 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.931 DEBUG:boto:Params: {} 2024-04-09T22:02:40.931 DEBUG:boto:Token: None 2024-04-09T22:02:40.931 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.931 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw= 2024-04-09T22:02:40.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw=', 'Content-Length': '0'} 2024-04-09T22:02:40.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f074c7a94402291-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.933 DEBUG:boto:path=/xryqqk-15/dummy 2024-04-09T22:02:40.933 DEBUG:boto:auth_path=/xryqqk-15/dummy 2024-04-09T22:02:40.933 DEBUG:boto:path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.933 DEBUG:boto:auth_path=/xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.933 DEBUG:boto:Method: GET 2024-04-09T22:02:40.933 DEBUG:boto:Path: /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.933 DEBUG:boto:Data: 2024-04-09T22:02:40.934 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.934 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.934 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.934 DEBUG:boto:Params: {} 2024-04-09T22:02:40.934 DEBUG:boto:Token: None 2024-04-09T22:02:40.934 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/dummy?versionId=kRroa4vyZJOJjSCIentxh7cKVwENfyG 2024-04-09T22:02:40.935 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw= 2024-04-09T22:02:40.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8eD4z0hr37/9Q/FjQRSt7ITpBvw=', 'Content-Length': '0'} 2024-04-09T22:02:40.937 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:59:29 GMT'), ('x-amz-version-id', 'kRroa4vyZJOJjSCIentxh7cKVwENfyG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000092e7215ebce876d-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.938 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.938 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.938 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.938 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.938 DEBUG:boto:Method: GET 2024-04-09T22:02:40.938 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.938 DEBUG:boto:Data: 2024-04-09T22:02:40.938 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.938 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.938 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.939 DEBUG:boto:Params: {} 2024-04-09T22:02:40.939 DEBUG:boto:Token: None 2024-04-09T22:02:40.939 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.939 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0= 2024-04-09T22:02:40.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0=', 'Content-Length': '0'} 2024-04-09T22:02:40.941 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009186166f2e3fa9e5-006615bb00-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.941 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.942 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.942 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.942 DEBUG:boto:path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.942 DEBUG:boto:auth_path=/xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.942 DEBUG:boto:Method: GET 2024-04-09T22:02:40.942 DEBUG:boto:Path: /xryqqk-15/?prefix=dummy 2024-04-09T22:02:40.942 DEBUG:boto:Data: 2024-04-09T22:02:40.943 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.943 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.943 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.943 DEBUG:boto:Params: {} 2024-04-09T22:02:40.943 DEBUG:boto:Token: None 2024-04-09T22:02:40.943 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.943 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0= 2024-04-09T22:02:40.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0=', 'Content-Length': '0'} 2024-04-09T22:02:40.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1e5d8e12a6f2b6a-006615bb00-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.946 DEBUG:boto:b'xryqqk-15dummy1000falsedummy2024-04-09T21:59:29.677Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:40.946 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.946 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.946 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.946 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.946 DEBUG:boto:Method: GET 2024-04-09T22:02:40.947 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.947 DEBUG:boto:Data: 2024-04-09T22:02:40.947 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.947 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.947 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.947 DEBUG:boto:Params: {} 2024-04-09T22:02:40.947 DEBUG:boto:Token: None 2024-04-09T22:02:40.947 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.947 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU= 2024-04-09T22:02:40.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU=', 'Content-Length': '0'} 2024-04-09T22:02:40.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000771e51767274f22e-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.950 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.950 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.950 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.950 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.951 DEBUG:boto:Method: GET 2024-04-09T22:02:40.951 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.951 DEBUG:boto:Data: 2024-04-09T22:02:40.951 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.951 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.951 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.951 DEBUG:boto:Params: {} 2024-04-09T22:02:40.951 DEBUG:boto:Token: None 2024-04-09T22:02:40.951 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.951 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU= 2024-04-09T22:02:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU=', 'Content-Length': '0'} 2024-04-09T22:02:40.953 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009deed92fe030c56a-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.954 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.954 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.954 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.954 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.954 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.954 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.954 DEBUG:boto:Data: 2024-04-09T22:02:40.954 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.954 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.955 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.955 DEBUG:boto:Params: {} 2024-04-09T22:02:40.955 DEBUG:boto:Token: None 2024-04-09T22:02:40.955 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.955 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ORXO9az26/4D7pLcK4cpwdwRz2A= 2024-04-09T22:02:40.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ORXO9az26/4D7pLcK4cpwdwRz2A=', 'Content-Length': '0'} 2024-04-09T22:02:40.957 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bfc56c08755c65da-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.957 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.957 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.957 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.957 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.957 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.957 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.958 DEBUG:boto:Data: 2024-04-09T22:02:40.958 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.958 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.958 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.958 DEBUG:boto:Params: {} 2024-04-09T22:02:40.958 DEBUG:boto:Token: None 2024-04-09T22:02:40.958 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.958 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ORXO9az26/4D7pLcK4cpwdwRz2A= 2024-04-09T22:02:40.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ORXO9az26/4D7pLcK4cpwdwRz2A=', 'Content-Length': '0'} 2024-04-09T22:02:40.960 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048faad315f17b643-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:40.961 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.961 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.961 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.962 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.962 DEBUG:boto:Method: GET 2024-04-09T22:02:40.962 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.962 DEBUG:boto:Data: 2024-04-09T22:02:40.962 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.962 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.963 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.963 DEBUG:boto:Params: {} 2024-04-09T22:02:40.963 DEBUG:boto:Token: None 2024-04-09T22:02:40.963 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.963 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU= 2024-04-09T22:02:40.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU=', 'Content-Length': '0'} 2024-04-09T22:02:40.965 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000298574a84b465e72-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.965 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.966 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.966 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.966 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.966 DEBUG:boto:Method: GET 2024-04-09T22:02:40.966 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.966 DEBUG:boto:Data: 2024-04-09T22:02:40.966 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.966 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.966 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.966 DEBUG:boto:Params: {} 2024-04-09T22:02:40.967 DEBUG:boto:Token: None 2024-04-09T22:02:40.967 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.967 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU= 2024-04-09T22:02:40.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU=', 'Content-Length': '0'} 2024-04-09T22:02:40.969 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000315fb4dfbf399eb1-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.969 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.970 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.970 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:40.970 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:40.970 DEBUG:boto:Method: GET 2024-04-09T22:02:40.970 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:40.970 DEBUG:boto:Data: 2024-04-09T22:02:40.970 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.970 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.970 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.970 DEBUG:boto:Params: {} 2024-04-09T22:02:40.971 DEBUG:boto:Token: None 2024-04-09T22:02:40.971 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.971 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0= 2024-04-09T22:02:40.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0=', 'Content-Length': '0'} 2024-04-09T22:02:40.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7bc033c3204632d-006615bb00-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.974 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.974 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:40.974 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:40.974 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:40.974 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:40.975 DEBUG:boto:Method: GET 2024-04-09T22:02:40.975 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:40.975 DEBUG:boto:Data: 2024-04-09T22:02:40.975 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.975 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.975 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.975 DEBUG:boto:Params: {} 2024-04-09T22:02:40.976 DEBUG:boto:Token: None 2024-04-09T22:02:40.976 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/ 2024-04-09T22:02:40.976 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0= 2024-04-09T22:02:40.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n+e1wz/DTh/H3vv0lY0Q5EDpOh0=', 'Content-Length': '0'} 2024-04-09T22:02:40.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a37bd17f9e03d11-006615bb00-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.979 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:40.979 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.980 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.980 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.980 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:40.980 DEBUG:boto:Data: 2024-04-09T22:02:40.980 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.980 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.981 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.981 DEBUG:boto:Params: {} 2024-04-09T22:02:40.981 DEBUG:boto:Token: None 2024-04-09T22:02:40.981 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj 2024-04-09T22:02:40.982 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q/zw/d5qDivqfKTJKbXkRSzMvHc= 2024-04-09T22:02:40.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q/zw/d5qDivqfKTJKbXkRSzMvHc=', 'Content-Length': '0'} 2024-04-09T22:02:40.985 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c662e935f5f3f81-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.985 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.986 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.986 DEBUG:boto:Method: HEAD 2024-04-09T22:02:40.986 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:40.986 DEBUG:boto:Data: 2024-04-09T22:02:40.987 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.987 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.987 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.987 DEBUG:boto:Params: {} 2024-04-09T22:02:40.987 DEBUG:boto:Token: None 2024-04-09T22:02:40.988 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj 2024-04-09T22:02:40.988 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q/zw/d5qDivqfKTJKbXkRSzMvHc= 2024-04-09T22:02:40.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q/zw/d5qDivqfKTJKbXkRSzMvHc=', 'Content-Length': '0'} 2024-04-09T22:02:40.991 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2847aa40510b76b-006615bb00-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:40.992 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.992 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.992 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.992 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.993 DEBUG:boto:Method: GET 2024-04-09T22:02:40.993 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.993 DEBUG:boto:Data: 2024-04-09T22:02:40.993 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.993 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:40.993 DEBUG:boto:Port: 8001 2024-04-09T22:02:40.994 DEBUG:boto:Params: {} 2024-04-09T22:02:40.994 DEBUG:boto:Token: None 2024-04-09T22:02:40.994 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:40 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.994 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU= 2024-04-09T22:02:40.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QhaxGMOyi5tg20pDQ9DnViwbLAU=', 'Content-Length': '0'} 2024-04-09T22:02:40.997 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074d357d56b0b644a-006615bb00-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:40.997 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:40.997 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:40.998 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.998 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.998 DEBUG:boto:Method: GET 2024-04-09T22:02:40.998 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:40.999 DEBUG:boto:Data: 2024-04-09T22:02:40.999 DEBUG:boto:Headers: {} 2024-04-09T22:02:40.999 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:40.999 DEBUG:boto:Port: 8000 2024-04-09T22:02:40.999 DEBUG:boto:Params: {} 2024-04-09T22:02:41.000 DEBUG:boto:Token: None 2024-04-09T22:02:41.000 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.000 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d86c5497fe9daa6b-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.042 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.043 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.043 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.043 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.043 DEBUG:boto:Method: GET 2024-04-09T22:02:41.043 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.043 DEBUG:boto:Data: 2024-04-09T22:02:41.043 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.043 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.044 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.044 DEBUG:boto:Params: {} 2024-04-09T22:02:41.044 DEBUG:boto:Token: None 2024-04-09T22:02:41.044 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1f4bd1a2dbe678-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.047 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.047 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.048 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.048 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.048 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.048 DEBUG:boto:Method: GET 2024-04-09T22:02:41.048 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.048 DEBUG:boto:Data: 2024-04-09T22:02:41.048 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.048 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.048 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.048 DEBUG:boto:Params: {} 2024-04-09T22:02:41.049 DEBUG:boto:Token: None 2024-04-09T22:02:41.049 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.049 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001cb5f0596698b94-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.051 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.052 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.052 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.052 DEBUG:boto:path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.052 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.052 DEBUG:boto:Method: GET 2024-04-09T22:02:41.052 DEBUG:boto:Path: /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.052 DEBUG:boto:Data: 2024-04-09T22:02:41.052 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.052 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.053 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.053 DEBUG:boto:Params: {} 2024-04-09T22:02:41.053 DEBUG:boto:Token: None 2024-04-09T22:02:41.053 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.053 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8= 2024-04-09T22:02:41.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8=', 'Content-Length': '0'} 2024-04-09T22:02:41.055 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000366f83264c5957c3-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.098 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.099 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.099 DEBUG:boto:path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.099 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.099 DEBUG:boto:Method: GET 2024-04-09T22:02:41.099 DEBUG:boto:Path: /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.099 DEBUG:boto:Data: 2024-04-09T22:02:41.099 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.099 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.099 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.099 DEBUG:boto:Params: {} 2024-04-09T22:02:41.099 DEBUG:boto:Token: None 2024-04-09T22:02:41.100 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.100 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8= 2024-04-09T22:02:41.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8=', 'Content-Length': '0'} 2024-04-09T22:02:41.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d7bb6167bf05cdd6-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.102 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.102 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.102 DEBUG:boto:path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.103 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.103 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.103 DEBUG:boto:Path: /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.103 DEBUG:boto:Data: 2024-04-09T22:02:41.103 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.103 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.103 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.103 DEBUG:boto:Params: {} 2024-04-09T22:02:41.103 DEBUG:boto:Token: None 2024-04-09T22:02:41.103 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.103 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lgjvgQ+Cw/sF2uHtcJo/n8awM0s= 2024-04-09T22:02:41.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lgjvgQ+Cw/sF2uHtcJo/n8awM0s=', 'Content-Length': '0'} 2024-04-09T22:02:41.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000575750d4ed9bbcc1-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.105 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.105 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.106 DEBUG:boto:path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.106 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.106 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.106 DEBUG:boto:Path: /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.106 DEBUG:boto:Data: 2024-04-09T22:02:41.106 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.106 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.106 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.106 DEBUG:boto:Params: {} 2024-04-09T22:02:41.106 DEBUG:boto:Token: None 2024-04-09T22:02:41.107 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.107 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lgjvgQ+Cw/sF2uHtcJo/n8awM0s= 2024-04-09T22:02:41.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lgjvgQ+Cw/sF2uHtcJo/n8awM0s=', 'Content-Length': '0'} 2024-04-09T22:02:41.108 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0a562bdb15845a8-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.109 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.109 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.109 DEBUG:boto:path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.109 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.109 DEBUG:boto:Method: GET 2024-04-09T22:02:41.109 DEBUG:boto:Path: /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.109 DEBUG:boto:Data: 2024-04-09T22:02:41.109 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.110 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.110 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.110 DEBUG:boto:Params: {} 2024-04-09T22:02:41.110 DEBUG:boto:Token: None 2024-04-09T22:02:41.110 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.110 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8= 2024-04-09T22:02:41.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8=', 'Content-Length': '0'} 2024-04-09T22:02:41.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056d8c5100781864d-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.112 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.112 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.112 DEBUG:boto:path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.112 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.112 DEBUG:boto:Method: GET 2024-04-09T22:02:41.112 DEBUG:boto:Path: /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.112 DEBUG:boto:Data: 2024-04-09T22:02:41.113 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.113 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.113 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.113 DEBUG:boto:Params: {} 2024-04-09T22:02:41.113 DEBUG:boto:Token: None 2024-04-09T22:02:41.113 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=WlVOuOCaDVTTkuos9UkExpZN5xD4yMo 2024-04-09T22:02:41.113 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8= 2024-04-09T22:02:41.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PBIrqLZLW0jt2tr4T9PYgdCcge8=', 'Content-Length': '0'} 2024-04-09T22:02:41.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'WlVOuOCaDVTTkuos9UkExpZN5xD4yMo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000620ff69683cdf7d5-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.158 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.159 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.159 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.159 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.159 DEBUG:boto:Method: GET 2024-04-09T22:02:41.159 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.159 DEBUG:boto:Data: 2024-04-09T22:02:41.159 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.159 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.159 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.159 DEBUG:boto:Params: {} 2024-04-09T22:02:41.160 DEBUG:boto:Token: None 2024-04-09T22:02:41.160 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.160 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd7107ccd9b6d73-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.163 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.164 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.164 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.164 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.165 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.165 DEBUG:boto:Method: GET 2024-04-09T22:02:41.165 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.165 DEBUG:boto:Data: 2024-04-09T22:02:41.165 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.166 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.166 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.166 DEBUG:boto:Params: {} 2024-04-09T22:02:41.166 DEBUG:boto:Token: None 2024-04-09T22:02:41.167 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.167 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c2331a9a496b7a7-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.170 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.171 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.171 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.171 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.172 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.172 DEBUG:boto:Data: 2024-04-09T22:02:41.172 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.172 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.172 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.172 DEBUG:boto:Params: {} 2024-04-09T22:02:41.173 DEBUG:boto:Token: None 2024-04-09T22:02:41.173 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.173 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a77ac9ee3ed9fc2-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.178 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.178 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.178 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.178 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.179 DEBUG:boto:Data: 2024-04-09T22:02:41.179 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.179 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.179 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.179 DEBUG:boto:Params: {} 2024-04-09T22:02:41.180 DEBUG:boto:Token: None 2024-04-09T22:02:41.180 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.180 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5a4480ef25f386e-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.184 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.185 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.185 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.185 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.185 DEBUG:boto:Method: GET 2024-04-09T22:02:41.185 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.186 DEBUG:boto:Data: 2024-04-09T22:02:41.186 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.186 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.186 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.186 DEBUG:boto:Params: {} 2024-04-09T22:02:41.187 DEBUG:boto:Token: None 2024-04-09T22:02:41.187 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.187 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8dd152c91c1b5c4-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.190 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.190 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.191 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.191 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.191 DEBUG:boto:Method: GET 2024-04-09T22:02:41.191 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.191 DEBUG:boto:Data: 2024-04-09T22:02:41.191 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.191 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.191 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.192 DEBUG:boto:Params: {} 2024-04-09T22:02:41.192 DEBUG:boto:Token: None 2024-04-09T22:02:41.192 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.192 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.194 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d319351ceb7eb4c-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.234 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.235 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.235 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.235 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.235 DEBUG:boto:Method: GET 2024-04-09T22:02:41.235 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.235 DEBUG:boto:Data: 2024-04-09T22:02:41.235 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.235 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.235 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.235 DEBUG:boto:Params: {} 2024-04-09T22:02:41.236 DEBUG:boto:Token: None 2024-04-09T22:02:41.236 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.236 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000809c6c1538884c88-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.238 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.239 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.239 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.239 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.239 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.239 DEBUG:boto:Method: GET 2024-04-09T22:02:41.239 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.239 DEBUG:boto:Data: 2024-04-09T22:02:41.239 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.240 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.240 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.240 DEBUG:boto:Params: {} 2024-04-09T22:02:41.240 DEBUG:boto:Token: None 2024-04-09T22:02:41.240 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c67f74571963823e-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.243 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.243 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.243 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.243 DEBUG:boto:path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.244 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.244 DEBUG:boto:Method: GET 2024-04-09T22:02:41.244 DEBUG:boto:Path: /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.244 DEBUG:boto:Data: 2024-04-09T22:02:41.244 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.244 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.244 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.244 DEBUG:boto:Params: {} 2024-04-09T22:02:41.244 DEBUG:boto:Token: None 2024-04-09T22:02:41.244 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.244 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ= 2024-04-09T22:02:41.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ=', 'Content-Length': '0'} 2024-04-09T22:02:41.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000329f243191868207-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.294 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.295 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.295 DEBUG:boto:path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.295 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.295 DEBUG:boto:Method: GET 2024-04-09T22:02:41.295 DEBUG:boto:Path: /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.295 DEBUG:boto:Data: 2024-04-09T22:02:41.295 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.295 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.295 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.295 DEBUG:boto:Params: {} 2024-04-09T22:02:41.296 DEBUG:boto:Token: None 2024-04-09T22:02:41.296 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.296 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ= 2024-04-09T22:02:41.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ=', 'Content-Length': '0'} 2024-04-09T22:02:41.298 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089551402bbd768a1-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.298 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.298 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.299 DEBUG:boto:path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.299 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.299 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.299 DEBUG:boto:Path: /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.299 DEBUG:boto:Data: 2024-04-09T22:02:41.299 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.299 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.299 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.299 DEBUG:boto:Params: {} 2024-04-09T22:02:41.299 DEBUG:boto:Token: None 2024-04-09T22:02:41.299 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.300 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RAfdvTm58B2TcVkxjJ7nwXncF5c= 2024-04-09T22:02:41.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RAfdvTm58B2TcVkxjJ7nwXncF5c=', 'Content-Length': '0'} 2024-04-09T22:02:41.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090af1171e88ac0e7-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.302 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.302 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.302 DEBUG:boto:path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.302 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.302 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.302 DEBUG:boto:Path: /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.302 DEBUG:boto:Data: 2024-04-09T22:02:41.303 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.303 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.303 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.303 DEBUG:boto:Params: {} 2024-04-09T22:02:41.303 DEBUG:boto:Token: None 2024-04-09T22:02:41.303 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.303 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RAfdvTm58B2TcVkxjJ7nwXncF5c= 2024-04-09T22:02:41.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RAfdvTm58B2TcVkxjJ7nwXncF5c=', 'Content-Length': '0'} 2024-04-09T22:02:41.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c98423375e3af3bc-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.305 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.305 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.306 DEBUG:boto:path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.306 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.306 DEBUG:boto:Method: GET 2024-04-09T22:02:41.306 DEBUG:boto:Path: /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.306 DEBUG:boto:Data: 2024-04-09T22:02:41.306 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.306 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.306 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.306 DEBUG:boto:Params: {} 2024-04-09T22:02:41.306 DEBUG:boto:Token: None 2024-04-09T22:02:41.307 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.307 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ= 2024-04-09T22:02:41.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ=', 'Content-Length': '0'} 2024-04-09T22:02:41.308 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf5d500990f1125f-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.309 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.309 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.309 DEBUG:boto:path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.309 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.309 DEBUG:boto:Method: GET 2024-04-09T22:02:41.309 DEBUG:boto:Path: /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.309 DEBUG:boto:Data: 2024-04-09T22:02:41.309 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.309 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.309 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.310 DEBUG:boto:Params: {} 2024-04-09T22:02:41.310 DEBUG:boto:Token: None 2024-04-09T22:02:41.310 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=JduizZozTy1Fs0rff.ccLhHDDTufRNy 2024-04-09T22:02:41.310 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ= 2024-04-09T22:02:41.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tXUcNXGmN4MhHX5+rDsLE7oSmmQ=', 'Content-Length': '0'} 2024-04-09T22:02:41.311 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'JduizZozTy1Fs0rff.ccLhHDDTufRNy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008e7226ea16eb5b4b-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.312 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.312 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.312 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.312 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.312 DEBUG:boto:Method: GET 2024-04-09T22:02:41.312 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.312 DEBUG:boto:Data: 2024-04-09T22:02:41.312 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.313 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.313 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.313 DEBUG:boto:Params: {} 2024-04-09T22:02:41.313 DEBUG:boto:Token: None 2024-04-09T22:02:41.313 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.313 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093a31693a1b4641d-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.316 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.316 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.316 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.316 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.316 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.317 DEBUG:boto:Method: GET 2024-04-09T22:02:41.317 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.317 DEBUG:boto:Data: 2024-04-09T22:02:41.317 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.317 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.317 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.317 DEBUG:boto:Params: {} 2024-04-09T22:02:41.317 DEBUG:boto:Token: None 2024-04-09T22:02:41.317 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.317 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069fb40d534cf7185-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.320 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.320 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.321 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.321 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.321 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.321 DEBUG:boto:Data: 2024-04-09T22:02:41.321 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.321 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.321 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.321 DEBUG:boto:Params: {} 2024-04-09T22:02:41.321 DEBUG:boto:Token: None 2024-04-09T22:02:41.321 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.322 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.325 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d05b5e98fbd5096f-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.325 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.325 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.325 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.325 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.325 DEBUG:boto:Data: 2024-04-09T22:02:41.325 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.325 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.325 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.326 DEBUG:boto:Params: {} 2024-04-09T22:02:41.326 DEBUG:boto:Token: None 2024-04-09T22:02:41.326 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.326 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f66b4c92233c09b-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.329 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.330 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.330 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.330 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.330 DEBUG:boto:Method: GET 2024-04-09T22:02:41.330 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.330 DEBUG:boto:Data: 2024-04-09T22:02:41.330 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.330 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.330 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.330 DEBUG:boto:Params: {} 2024-04-09T22:02:41.331 DEBUG:boto:Token: None 2024-04-09T22:02:41.331 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.331 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cda159f701c4471-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.374 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.375 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.375 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.375 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.375 DEBUG:boto:Method: GET 2024-04-09T22:02:41.375 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.375 DEBUG:boto:Data: 2024-04-09T22:02:41.375 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.375 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.375 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.375 DEBUG:boto:Params: {} 2024-04-09T22:02:41.376 DEBUG:boto:Token: None 2024-04-09T22:02:41.376 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.376 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.378 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000043d306d997829389-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.378 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.378 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.379 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.379 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.379 DEBUG:boto:Method: GET 2024-04-09T22:02:41.379 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.379 DEBUG:boto:Data: 2024-04-09T22:02:41.379 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.379 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.379 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.379 DEBUG:boto:Params: {} 2024-04-09T22:02:41.379 DEBUG:boto:Token: None 2024-04-09T22:02:41.380 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.380 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b52d8adead3d087e-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.383 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.383 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.383 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.383 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.383 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.384 DEBUG:boto:Method: GET 2024-04-09T22:02:41.384 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.384 DEBUG:boto:Data: 2024-04-09T22:02:41.384 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.384 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.384 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.384 DEBUG:boto:Params: {} 2024-04-09T22:02:41.384 DEBUG:boto:Token: None 2024-04-09T22:02:41.384 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.384 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025c06f54629a5e3-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.387 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.387 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.387 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.387 DEBUG:boto:path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.387 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.387 DEBUG:boto:Method: GET 2024-04-09T22:02:41.388 DEBUG:boto:Path: /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.388 DEBUG:boto:Data: 2024-04-09T22:02:41.388 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.388 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.388 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.388 DEBUG:boto:Params: {} 2024-04-09T22:02:41.388 DEBUG:boto:Token: None 2024-04-09T22:02:41.388 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.388 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc= 2024-04-09T22:02:41.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc=', 'Content-Length': '0'} 2024-04-09T22:02:41.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebb34ce0622ede4c-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.390 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.390 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.390 DEBUG:boto:path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.390 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.391 DEBUG:boto:Method: GET 2024-04-09T22:02:41.391 DEBUG:boto:Path: /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.391 DEBUG:boto:Data: 2024-04-09T22:02:41.391 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.391 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.391 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.391 DEBUG:boto:Params: {} 2024-04-09T22:02:41.391 DEBUG:boto:Token: None 2024-04-09T22:02:41.391 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.391 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc= 2024-04-09T22:02:41.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc=', 'Content-Length': '0'} 2024-04-09T22:02:41.393 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c55ab4c276c18bd-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.438 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.438 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.439 DEBUG:boto:path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.439 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.439 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.439 DEBUG:boto:Path: /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.439 DEBUG:boto:Data: 2024-04-09T22:02:41.439 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.439 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.439 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.439 DEBUG:boto:Params: {} 2024-04-09T22:02:41.440 DEBUG:boto:Token: None 2024-04-09T22:02:41.440 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.440 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EotTvyVUewXjg81DtW9p55x6u+4= 2024-04-09T22:02:41.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EotTvyVUewXjg81DtW9p55x6u+4=', 'Content-Length': '0'} 2024-04-09T22:02:41.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bed73951cc8e58a5-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.442 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.442 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.443 DEBUG:boto:path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.443 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.443 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.443 DEBUG:boto:Path: /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.443 DEBUG:boto:Data: 2024-04-09T22:02:41.443 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.443 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.443 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.443 DEBUG:boto:Params: {} 2024-04-09T22:02:41.443 DEBUG:boto:Token: None 2024-04-09T22:02:41.443 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.444 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EotTvyVUewXjg81DtW9p55x6u+4= 2024-04-09T22:02:41.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EotTvyVUewXjg81DtW9p55x6u+4=', 'Content-Length': '0'} 2024-04-09T22:02:41.445 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2652879b4f69362-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.446 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.446 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.446 DEBUG:boto:path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.446 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.446 DEBUG:boto:Method: GET 2024-04-09T22:02:41.446 DEBUG:boto:Path: /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.446 DEBUG:boto:Data: 2024-04-09T22:02:41.446 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.447 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.447 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.447 DEBUG:boto:Params: {} 2024-04-09T22:02:41.447 DEBUG:boto:Token: None 2024-04-09T22:02:41.447 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.447 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc= 2024-04-09T22:02:41.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc=', 'Content-Length': '0'} 2024-04-09T22:02:41.449 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e5ab3e33045cb0c-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.449 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.449 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.449 DEBUG:boto:path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.449 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.449 DEBUG:boto:Method: GET 2024-04-09T22:02:41.450 DEBUG:boto:Path: /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.450 DEBUG:boto:Data: 2024-04-09T22:02:41.450 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.450 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.450 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.450 DEBUG:boto:Params: {} 2024-04-09T22:02:41.450 DEBUG:boto:Token: None 2024-04-09T22:02:41.450 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=TBks51TGviAJ2FKkudDkJCF6TK7A.M1 2024-04-09T22:02:41.450 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc= 2024-04-09T22:02:41.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q+X3CwqZUAY9UFW0GgCT6dJ3xpc=', 'Content-Length': '0'} 2024-04-09T22:02:41.452 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'TBks51TGviAJ2FKkudDkJCF6TK7A.M1'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006496f82654a0dc67-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.453 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.453 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.453 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.453 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.453 DEBUG:boto:Method: GET 2024-04-09T22:02:41.453 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.453 DEBUG:boto:Data: 2024-04-09T22:02:41.453 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.453 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.453 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.453 DEBUG:boto:Params: {} 2024-04-09T22:02:41.454 DEBUG:boto:Token: None 2024-04-09T22:02:41.454 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.454 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1c79acee7ee24ea-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.457 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.458 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.458 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.458 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.458 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.458 DEBUG:boto:Method: GET 2024-04-09T22:02:41.458 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.458 DEBUG:boto:Data: 2024-04-09T22:02:41.458 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.458 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.458 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.459 DEBUG:boto:Params: {} 2024-04-09T22:02:41.459 DEBUG:boto:Token: None 2024-04-09T22:02:41.459 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.459 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000019e939d9d608756-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.461 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.462 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.462 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.463 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.463 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.463 DEBUG:boto:Data: 2024-04-09T22:02:41.463 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.463 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.463 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.464 DEBUG:boto:Params: {} 2024-04-09T22:02:41.464 DEBUG:boto:Token: None 2024-04-09T22:02:41.464 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096535aba1b87cd80-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.468 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.468 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.469 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.469 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.469 DEBUG:boto:Data: 2024-04-09T22:02:41.469 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.469 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.470 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.470 DEBUG:boto:Params: {} 2024-04-09T22:02:41.470 DEBUG:boto:Token: None 2024-04-09T22:02:41.470 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c737ca32229b6549-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.474 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.475 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.475 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.475 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.475 DEBUG:boto:Method: GET 2024-04-09T22:02:41.476 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.476 DEBUG:boto:Data: 2024-04-09T22:02:41.476 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.476 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.476 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.477 DEBUG:boto:Params: {} 2024-04-09T22:02:41.477 DEBUG:boto:Token: None 2024-04-09T22:02:41.477 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.477 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.480 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019b330f39fd95e8b-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.522 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.523 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.523 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.523 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.523 DEBUG:boto:Method: GET 2024-04-09T22:02:41.523 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.523 DEBUG:boto:Data: 2024-04-09T22:02:41.523 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.523 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.523 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.524 DEBUG:boto:Params: {} 2024-04-09T22:02:41.524 DEBUG:boto:Token: None 2024-04-09T22:02:41.524 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.524 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000066b3d6a336d325b4-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.526 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.527 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.527 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.527 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.527 DEBUG:boto:Method: GET 2024-04-09T22:02:41.527 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.527 DEBUG:boto:Data: 2024-04-09T22:02:41.527 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.527 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.528 DEBUG:boto:Params: {} 2024-04-09T22:02:41.528 DEBUG:boto:Token: None 2024-04-09T22:02:41.528 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c60d49c00ca0a52-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.531 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.531 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.531 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.531 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.531 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.532 DEBUG:boto:Method: GET 2024-04-09T22:02:41.532 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.532 DEBUG:boto:Data: 2024-04-09T22:02:41.532 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.532 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.532 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.532 DEBUG:boto:Params: {} 2024-04-09T22:02:41.532 DEBUG:boto:Token: None 2024-04-09T22:02:41.532 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7e1b261858934ab-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.535 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.535 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.536 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.536 DEBUG:boto:path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.536 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.536 DEBUG:boto:Method: GET 2024-04-09T22:02:41.536 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.536 DEBUG:boto:Data: 2024-04-09T22:02:41.536 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.536 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.536 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.536 DEBUG:boto:Params: {} 2024-04-09T22:02:41.536 DEBUG:boto:Token: None 2024-04-09T22:02:41.537 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.537 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA= 2024-04-09T22:02:41.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA=', 'Content-Length': '0'} 2024-04-09T22:02:41.538 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d91f7313db12f62-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.539 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.539 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.539 DEBUG:boto:path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.539 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.539 DEBUG:boto:Method: GET 2024-04-09T22:02:41.539 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.539 DEBUG:boto:Data: 2024-04-09T22:02:41.539 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.539 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.540 DEBUG:boto:Params: {} 2024-04-09T22:02:41.540 DEBUG:boto:Token: None 2024-04-09T22:02:41.540 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA= 2024-04-09T22:02:41.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA=', 'Content-Length': '0'} 2024-04-09T22:02:41.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000327c5c883b4d500a-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.542 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.543 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.543 DEBUG:boto:path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.543 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.543 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.543 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.543 DEBUG:boto:Data: 2024-04-09T22:02:41.543 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.543 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.543 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.543 DEBUG:boto:Params: {} 2024-04-09T22:02:41.543 DEBUG:boto:Token: None 2024-04-09T22:02:41.544 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.544 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gkQxd+g5cyHhC6mnZzWgRI89jug= 2024-04-09T22:02:41.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gkQxd+g5cyHhC6mnZzWgRI89jug=', 'Content-Length': '0'} 2024-04-09T22:02:41.546 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd81256f7d804403-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.546 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.546 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.546 DEBUG:boto:path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.546 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.546 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.546 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.546 DEBUG:boto:Data: 2024-04-09T22:02:41.547 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.547 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.547 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.547 DEBUG:boto:Params: {} 2024-04-09T22:02:41.547 DEBUG:boto:Token: None 2024-04-09T22:02:41.547 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.547 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gkQxd+g5cyHhC6mnZzWgRI89jug= 2024-04-09T22:02:41.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gkQxd+g5cyHhC6mnZzWgRI89jug=', 'Content-Length': '0'} 2024-04-09T22:02:41.549 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024fc98b7f32af66c-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.549 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.549 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.550 DEBUG:boto:path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.550 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.550 DEBUG:boto:Method: GET 2024-04-09T22:02:41.550 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.550 DEBUG:boto:Data: 2024-04-09T22:02:41.550 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.550 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.550 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.550 DEBUG:boto:Params: {} 2024-04-09T22:02:41.550 DEBUG:boto:Token: None 2024-04-09T22:02:41.550 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.551 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA= 2024-04-09T22:02:41.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA=', 'Content-Length': '0'} 2024-04-09T22:02:41.552 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca0e0036122b115a-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.594 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.594 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.595 DEBUG:boto:path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.595 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.595 DEBUG:boto:Method: GET 2024-04-09T22:02:41.595 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.595 DEBUG:boto:Data: 2024-04-09T22:02:41.595 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.595 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.595 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.595 DEBUG:boto:Params: {} 2024-04-09T22:02:41.595 DEBUG:boto:Token: None 2024-04-09T22:02:41.596 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq 2024-04-09T22:02:41.596 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA= 2024-04-09T22:02:41.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:obLXuQVXEo0TWIO/9X4OjPElQKA=', 'Content-Length': '0'} 2024-04-09T22:02:41.598 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9i4vn5lmuYHa8oi6UyZDWS8fqdzaphq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042ed32bbb987e7f1-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.598 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.598 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.598 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.598 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.598 DEBUG:boto:Method: GET 2024-04-09T22:02:41.598 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.599 DEBUG:boto:Data: 2024-04-09T22:02:41.599 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.599 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.599 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.599 DEBUG:boto:Params: {} 2024-04-09T22:02:41.599 DEBUG:boto:Token: None 2024-04-09T22:02:41.599 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.599 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f6101d85616ec7-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.602 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.602 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.603 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.603 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.603 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.603 DEBUG:boto:Method: GET 2024-04-09T22:02:41.603 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.603 DEBUG:boto:Data: 2024-04-09T22:02:41.603 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.603 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.603 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.603 DEBUG:boto:Params: {} 2024-04-09T22:02:41.604 DEBUG:boto:Token: None 2024-04-09T22:02:41.604 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.604 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047328bbf7220b37b-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.606 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.607 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.607 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.607 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.607 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.607 DEBUG:boto:Data: 2024-04-09T22:02:41.607 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.607 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.607 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.608 DEBUG:boto:Params: {} 2024-04-09T22:02:41.608 DEBUG:boto:Token: None 2024-04-09T22:02:41.608 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.608 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.611 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ff46d8010b12d8b-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.611 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.612 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.612 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.612 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.612 DEBUG:boto:Data: 2024-04-09T22:02:41.612 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.612 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.612 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.612 DEBUG:boto:Params: {} 2024-04-09T22:02:41.612 DEBUG:boto:Token: None 2024-04-09T22:02:41.612 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.612 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d50c6a3beff6d45d-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.616 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.616 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.616 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.616 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.617 DEBUG:boto:Method: GET 2024-04-09T22:02:41.617 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.617 DEBUG:boto:Data: 2024-04-09T22:02:41.617 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.617 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.617 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.617 DEBUG:boto:Params: {} 2024-04-09T22:02:41.617 DEBUG:boto:Token: None 2024-04-09T22:02:41.617 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.617 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000096af3e1fbb74bb5b-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.666 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.667 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.667 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.667 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.667 DEBUG:boto:Method: GET 2024-04-09T22:02:41.667 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.667 DEBUG:boto:Data: 2024-04-09T22:02:41.667 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.667 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.667 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.667 DEBUG:boto:Params: {} 2024-04-09T22:02:41.668 DEBUG:boto:Token: None 2024-04-09T22:02:41.668 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.668 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.670 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d8f4740bce0e9b3-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.670 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.670 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.671 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.671 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.671 DEBUG:boto:Method: GET 2024-04-09T22:02:41.671 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.671 DEBUG:boto:Data: 2024-04-09T22:02:41.671 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.671 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.671 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.671 DEBUG:boto:Params: {} 2024-04-09T22:02:41.671 DEBUG:boto:Token: None 2024-04-09T22:02:41.671 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.672 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006a012e5d3f12f48-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.714 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.715 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.715 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.715 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.715 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.715 DEBUG:boto:Method: GET 2024-04-09T22:02:41.715 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.715 DEBUG:boto:Data: 2024-04-09T22:02:41.716 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.716 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.716 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.716 DEBUG:boto:Params: {} 2024-04-09T22:02:41.716 DEBUG:boto:Token: None 2024-04-09T22:02:41.716 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.716 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bfc1e241a4ec900-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.719 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.720 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.720 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.720 DEBUG:boto:path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.720 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.720 DEBUG:boto:Method: GET 2024-04-09T22:02:41.720 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.720 DEBUG:boto:Data: 2024-04-09T22:02:41.720 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.720 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.720 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.720 DEBUG:boto:Params: {} 2024-04-09T22:02:41.721 DEBUG:boto:Token: None 2024-04-09T22:02:41.721 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.721 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w= 2024-04-09T22:02:41.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w=', 'Content-Length': '0'} 2024-04-09T22:02:41.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b6f4bdf91ff81dd4-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.723 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.723 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.723 DEBUG:boto:path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.723 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.723 DEBUG:boto:Method: GET 2024-04-09T22:02:41.724 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.724 DEBUG:boto:Data: 2024-04-09T22:02:41.724 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.724 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.724 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.724 DEBUG:boto:Params: {} 2024-04-09T22:02:41.724 DEBUG:boto:Token: None 2024-04-09T22:02:41.724 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.724 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w= 2024-04-09T22:02:41.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w=', 'Content-Length': '0'} 2024-04-09T22:02:41.726 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000044642f5e89345990-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.726 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.727 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.727 DEBUG:boto:path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.727 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.727 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.727 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.727 DEBUG:boto:Data: 2024-04-09T22:02:41.727 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.727 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.727 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.727 DEBUG:boto:Params: {} 2024-04-09T22:02:41.727 DEBUG:boto:Token: None 2024-04-09T22:02:41.728 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.728 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fN4jTlBYIp/rzUMsPpnSAa495wk= 2024-04-09T22:02:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fN4jTlBYIp/rzUMsPpnSAa495wk=', 'Content-Length': '0'} 2024-04-09T22:02:41.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047184d817cb0fb74-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.730 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.730 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.730 DEBUG:boto:path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.730 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.730 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.730 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.730 DEBUG:boto:Data: 2024-04-09T22:02:41.730 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.731 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.731 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.731 DEBUG:boto:Params: {} 2024-04-09T22:02:41.731 DEBUG:boto:Token: None 2024-04-09T22:02:41.731 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.731 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fN4jTlBYIp/rzUMsPpnSAa495wk= 2024-04-09T22:02:41.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fN4jTlBYIp/rzUMsPpnSAa495wk=', 'Content-Length': '0'} 2024-04-09T22:02:41.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000ea34ba7caeb273a-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.733 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.733 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.733 DEBUG:boto:path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.733 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.733 DEBUG:boto:Method: GET 2024-04-09T22:02:41.734 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.734 DEBUG:boto:Data: 2024-04-09T22:02:41.734 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.734 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.734 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.734 DEBUG:boto:Params: {} 2024-04-09T22:02:41.734 DEBUG:boto:Token: None 2024-04-09T22:02:41.734 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.734 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w= 2024-04-09T22:02:41.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w=', 'Content-Length': '0'} 2024-04-09T22:02:41.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004af58d79d261651a-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.778 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.779 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.779 DEBUG:boto:path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.779 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.779 DEBUG:boto:Method: GET 2024-04-09T22:02:41.779 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.779 DEBUG:boto:Data: 2024-04-09T22:02:41.779 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.779 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.779 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.780 DEBUG:boto:Params: {} 2024-04-09T22:02:41.780 DEBUG:boto:Token: None 2024-04-09T22:02:41.780 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW 2024-04-09T22:02:41.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w= 2024-04-09T22:02:41.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mitFeSD5gVusWJ3mQPtqy+8Ca9w=', 'Content-Length': '0'} 2024-04-09T22:02:41.782 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9iw.TDO7FhgnqwpVBEInTJKvjjyNOlW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023d6a79a33d0e892-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.783 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.783 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.783 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.783 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.783 DEBUG:boto:Method: GET 2024-04-09T22:02:41.783 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.783 DEBUG:boto:Data: 2024-04-09T22:02:41.783 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.783 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.783 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.783 DEBUG:boto:Params: {} 2024-04-09T22:02:41.784 DEBUG:boto:Token: None 2024-04-09T22:02:41.784 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.784 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a89b0dc314df7d80-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.830 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.831 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.832 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.832 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.832 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.832 DEBUG:boto:Method: GET 2024-04-09T22:02:41.832 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.832 DEBUG:boto:Data: 2024-04-09T22:02:41.832 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.832 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.832 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.832 DEBUG:boto:Params: {} 2024-04-09T22:02:41.832 DEBUG:boto:Token: None 2024-04-09T22:02:41.833 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.833 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078b437551c6a419f-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.836 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.836 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.836 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.836 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.836 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.836 DEBUG:boto:Data: 2024-04-09T22:02:41.836 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.837 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.837 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.837 DEBUG:boto:Params: {} 2024-04-09T22:02:41.837 DEBUG:boto:Token: None 2024-04-09T22:02:41.837 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.837 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.840 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000341ab5be3ab9bd4f-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.841 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.841 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.841 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.841 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:41.841 DEBUG:boto:Data: 2024-04-09T22:02:41.841 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.841 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.841 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.841 DEBUG:boto:Params: {} 2024-04-09T22:02:41.842 DEBUG:boto:Token: None 2024-04-09T22:02:41.842 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj 2024-04-09T22:02:41.842 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0= 2024-04-09T22:02:41.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HRs5HuSs7jiJVbn+suncUeyOnw0=', 'Content-Length': '0'} 2024-04-09T22:02:41.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8189675cc1db0ba-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.846 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.846 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.846 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.846 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.846 DEBUG:boto:Method: GET 2024-04-09T22:02:41.846 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.846 DEBUG:boto:Data: 2024-04-09T22:02:41.846 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.846 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.846 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.847 DEBUG:boto:Params: {} 2024-04-09T22:02:41.847 DEBUG:boto:Token: None 2024-04-09T22:02:41.847 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.847 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.849 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004619b7b9ec749844-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.890 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.891 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.891 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.891 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.891 DEBUG:boto:Method: GET 2024-04-09T22:02:41.891 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.891 DEBUG:boto:Data: 2024-04-09T22:02:41.891 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.891 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.891 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.892 DEBUG:boto:Params: {} 2024-04-09T22:02:41.892 DEBUG:boto:Token: None 2024-04-09T22:02:41.892 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:41.892 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc= 2024-04-09T22:02:41.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iwHScoBmQ8UxreaNFoDNngxbwMc=', 'Content-Length': '0'} 2024-04-09T22:02:41.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fd4280423cdef0a-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.895 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.895 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.895 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.895 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.895 DEBUG:boto:Method: GET 2024-04-09T22:02:41.896 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.896 DEBUG:boto:Data: 2024-04-09T22:02:41.896 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.896 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.896 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.896 DEBUG:boto:Params: {} 2024-04-09T22:02:41.896 DEBUG:boto:Token: None 2024-04-09T22:02:41.896 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.897 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002abcc7f6f4e5e188-006615bb01-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.946 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.947 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:41.947 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:41.947 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.947 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:41.947 DEBUG:boto:Method: GET 2024-04-09T22:02:41.947 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:41.947 DEBUG:boto:Data: 2024-04-09T22:02:41.947 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.947 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.947 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.947 DEBUG:boto:Params: {} 2024-04-09T22:02:41.948 DEBUG:boto:Token: None 2024-04-09T22:02:41.948 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/ 2024-04-09T22:02:41.948 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I= 2024-04-09T22:02:41.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jJFt30hoYtJM5wkS8hs5dgfGi+I=', 'Content-Length': '0'} 2024-04-09T22:02:41.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df9319386670a9a6-006615bb01-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.951 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:41.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.951 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.951 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.951 DEBUG:boto:path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.951 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.952 DEBUG:boto:Method: GET 2024-04-09T22:02:41.952 DEBUG:boto:Path: /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.952 DEBUG:boto:Data: 2024-04-09T22:02:41.952 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.952 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.952 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.952 DEBUG:boto:Params: {} 2024-04-09T22:02:41.952 DEBUG:boto:Token: None 2024-04-09T22:02:41.952 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.952 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VzgTHN7GWfIKy4lC49GNxWT5l78= 2024-04-09T22:02:41.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VzgTHN7GWfIKy4lC49GNxWT5l78=', 'Content-Length': '0'} 2024-04-09T22:02:41.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005139532a502223b2-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.955 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.955 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.955 DEBUG:boto:path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.955 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.955 DEBUG:boto:Method: GET 2024-04-09T22:02:41.956 DEBUG:boto:Path: /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.956 DEBUG:boto:Data: 2024-04-09T22:02:41.956 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.956 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.956 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.957 DEBUG:boto:Params: {} 2024-04-09T22:02:41.957 DEBUG:boto:Token: None 2024-04-09T22:02:41.957 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.957 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VzgTHN7GWfIKy4lC49GNxWT5l78= 2024-04-09T22:02:41.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VzgTHN7GWfIKy4lC49GNxWT5l78=', 'Content-Length': '0'} 2024-04-09T22:02:41.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000005c646b4217ed39a-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.959 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.960 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.960 DEBUG:boto:path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.960 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.960 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.960 DEBUG:boto:Path: /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.960 DEBUG:boto:Data: 2024-04-09T22:02:41.960 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.960 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.960 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.960 DEBUG:boto:Params: {} 2024-04-09T22:02:41.961 DEBUG:boto:Token: None 2024-04-09T22:02:41.961 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.961 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3SsMYHz1BFfIQh5ip/asPqQfCzc= 2024-04-09T22:02:41.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3SsMYHz1BFfIQh5ip/asPqQfCzc=', 'Content-Length': '0'} 2024-04-09T22:02:41.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000023b13334fe6d943a-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.963 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.963 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.963 DEBUG:boto:path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.964 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.964 DEBUG:boto:Method: HEAD 2024-04-09T22:02:41.964 DEBUG:boto:Path: /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.964 DEBUG:boto:Data: 2024-04-09T22:02:41.964 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.964 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:41.964 DEBUG:boto:Port: 8000 2024-04-09T22:02:41.964 DEBUG:boto:Params: {} 2024-04-09T22:02:41.964 DEBUG:boto:Token: None 2024-04-09T22:02:41.964 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.965 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3SsMYHz1BFfIQh5ip/asPqQfCzc= 2024-04-09T22:02:41.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3SsMYHz1BFfIQh5ip/asPqQfCzc=', 'Content-Length': '0'} 2024-04-09T22:02:41.966 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035007da79271c294-006615bb01-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:41.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:41.967 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:41.967 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:41.967 DEBUG:boto:path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.967 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.967 DEBUG:boto:Method: GET 2024-04-09T22:02:41.967 DEBUG:boto:Path: /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.967 DEBUG:boto:Data: 2024-04-09T22:02:41.967 DEBUG:boto:Headers: {} 2024-04-09T22:02:41.967 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:41.967 DEBUG:boto:Port: 8001 2024-04-09T22:02:41.967 DEBUG:boto:Params: {} 2024-04-09T22:02:41.968 DEBUG:boto:Token: None 2024-04-09T22:02:41.968 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:41 GMT /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:41.968 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VzgTHN7GWfIKy4lC49GNxWT5l78= 2024-04-09T22:02:41.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VzgTHN7GWfIKy4lC49GNxWT5l78=', 'Content-Length': '0'} 2024-04-09T22:02:41.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5423e152aee6e5d-006615bb01-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.010 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.011 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.011 DEBUG:boto:path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:42.011 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:42.011 DEBUG:boto:Method: GET 2024-04-09T22:02:42.011 DEBUG:boto:Path: /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:42.011 DEBUG:boto:Data: 2024-04-09T22:02:42.011 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.011 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.011 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.011 DEBUG:boto:Params: {} 2024-04-09T22:02:42.011 DEBUG:boto:Token: None 2024-04-09T22:02:42.012 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=e0tEUBVITt61ncOaHmD8QxbssFQjO92 2024-04-09T22:02:42.012 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:h14oWrMx4sLYxtOsYvO5iYoDVcI= 2024-04-09T22:02:42.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h14oWrMx4sLYxtOsYvO5iYoDVcI=', 'Content-Length': '0'} 2024-04-09T22:02:42.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'e0tEUBVITt61ncOaHmD8QxbssFQjO92'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a45b6c484ae215e-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.058 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.059 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.059 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.059 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.059 DEBUG:boto:Method: GET 2024-04-09T22:02:42.060 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.060 DEBUG:boto:Data: 2024-04-09T22:02:42.060 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.060 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.060 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.061 DEBUG:boto:Params: {} 2024-04-09T22:02:42.061 DEBUG:boto:Token: None 2024-04-09T22:02:42.061 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.062 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9dd39d9ceb73ea-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.065 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.066 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.066 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.066 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.066 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.066 DEBUG:boto:Method: GET 2024-04-09T22:02:42.067 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.067 DEBUG:boto:Data: 2024-04-09T22:02:42.067 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.067 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.067 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.068 DEBUG:boto:Params: {} 2024-04-09T22:02:42.068 DEBUG:boto:Token: None 2024-04-09T22:02:42.068 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.068 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d36789091ac46d-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.071 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.072 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.072 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.072 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.073 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.073 DEBUG:boto:Data: 2024-04-09T22:02:42.073 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.073 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.073 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.073 DEBUG:boto:Params: {} 2024-04-09T22:02:42.073 DEBUG:boto:Token: None 2024-04-09T22:02:42.074 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.074 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.077 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0b01b98510a1812-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.078 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.078 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.078 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.078 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.078 DEBUG:boto:Data: 2024-04-09T22:02:42.078 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.078 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.078 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.078 DEBUG:boto:Params: {} 2024-04-09T22:02:42.079 DEBUG:boto:Token: None 2024-04-09T22:02:42.080 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.080 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.083 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001fdb04764898d36-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.083 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.083 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.083 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.084 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.084 DEBUG:boto:Method: GET 2024-04-09T22:02:42.084 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.084 DEBUG:boto:Data: 2024-04-09T22:02:42.084 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.084 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.084 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.084 DEBUG:boto:Params: {} 2024-04-09T22:02:42.084 DEBUG:boto:Token: None 2024-04-09T22:02:42.084 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.085 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000461c073e9bd06213-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.130 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.131 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.131 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.131 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.132 DEBUG:boto:Method: GET 2024-04-09T22:02:42.132 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.132 DEBUG:boto:Data: 2024-04-09T22:02:42.132 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.132 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.133 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.133 DEBUG:boto:Params: {} 2024-04-09T22:02:42.133 DEBUG:boto:Token: None 2024-04-09T22:02:42.134 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.134 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000952d9aaa75565958-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.137 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.137 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.138 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.138 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.138 DEBUG:boto:Method: GET 2024-04-09T22:02:42.138 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.138 DEBUG:boto:Data: 2024-04-09T22:02:42.139 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.139 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.139 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.139 DEBUG:boto:Params: {} 2024-04-09T22:02:42.139 DEBUG:boto:Token: None 2024-04-09T22:02:42.140 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.140 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f60b0328c1d0b35d-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.143 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.144 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.144 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.144 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.144 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.144 DEBUG:boto:Method: GET 2024-04-09T22:02:42.144 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.145 DEBUG:boto:Data: 2024-04-09T22:02:42.145 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.145 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.145 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.145 DEBUG:boto:Params: {} 2024-04-09T22:02:42.145 DEBUG:boto:Token: None 2024-04-09T22:02:42.145 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.146 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000867721c454b9641f-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.148 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.149 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.149 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.149 DEBUG:boto:path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.149 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.149 DEBUG:boto:Method: GET 2024-04-09T22:02:42.149 DEBUG:boto:Path: /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.149 DEBUG:boto:Data: 2024-04-09T22:02:42.149 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.149 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.149 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.150 DEBUG:boto:Params: {} 2024-04-09T22:02:42.150 DEBUG:boto:Token: None 2024-04-09T22:02:42.150 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.150 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE= 2024-04-09T22:02:42.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.152 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000caaeaf767a4bbc24-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.152 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.152 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.152 DEBUG:boto:path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.152 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.152 DEBUG:boto:Method: GET 2024-04-09T22:02:42.153 DEBUG:boto:Path: /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.153 DEBUG:boto:Data: 2024-04-09T22:02:42.153 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.153 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.153 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.153 DEBUG:boto:Params: {} 2024-04-09T22:02:42.153 DEBUG:boto:Token: None 2024-04-09T22:02:42.153 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.153 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE= 2024-04-09T22:02:42.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be355dc14a69eeba-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.155 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.156 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.156 DEBUG:boto:path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.156 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.156 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.156 DEBUG:boto:Path: /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.156 DEBUG:boto:Data: 2024-04-09T22:02:42.156 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.156 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.156 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.156 DEBUG:boto:Params: {} 2024-04-09T22:02:42.156 DEBUG:boto:Token: None 2024-04-09T22:02:42.157 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.157 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JGg/R5dptMHAUpsj6xcPFttmCAM= 2024-04-09T22:02:42.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JGg/R5dptMHAUpsj6xcPFttmCAM=', 'Content-Length': '0'} 2024-04-09T22:02:42.159 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e1ff9a591fe7c89-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.159 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.159 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.159 DEBUG:boto:path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.159 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.159 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.159 DEBUG:boto:Path: /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.159 DEBUG:boto:Data: 2024-04-09T22:02:42.159 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.160 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.160 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.160 DEBUG:boto:Params: {} 2024-04-09T22:02:42.160 DEBUG:boto:Token: None 2024-04-09T22:02:42.160 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.160 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JGg/R5dptMHAUpsj6xcPFttmCAM= 2024-04-09T22:02:42.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JGg/R5dptMHAUpsj6xcPFttmCAM=', 'Content-Length': '0'} 2024-04-09T22:02:42.162 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000091deee20be4e19b5-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.162 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.162 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.162 DEBUG:boto:path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.162 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.163 DEBUG:boto:Method: GET 2024-04-09T22:02:42.163 DEBUG:boto:Path: /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.163 DEBUG:boto:Data: 2024-04-09T22:02:42.163 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.163 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.163 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.163 DEBUG:boto:Params: {} 2024-04-09T22:02:42.163 DEBUG:boto:Token: None 2024-04-09T22:02:42.163 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.163 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE= 2024-04-09T22:02:42.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.165 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b15ce242cca4737c-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.210 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.211 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.211 DEBUG:boto:path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.211 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.211 DEBUG:boto:Method: GET 2024-04-09T22:02:42.211 DEBUG:boto:Path: /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.211 DEBUG:boto:Data: 2024-04-09T22:02:42.211 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.211 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.211 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.211 DEBUG:boto:Params: {} 2024-04-09T22:02:42.212 DEBUG:boto:Token: None 2024-04-09T22:02:42.212 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3 2024-04-09T22:02:42.212 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE= 2024-04-09T22:02:42.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L80qF3gUSKXAdrzfblNTo93RZLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.215 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'qS-jaSb9pCWrKjwAQ6neHckrgYDdBl3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a441ec5236fefb60-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.215 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.215 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.215 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.215 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.216 DEBUG:boto:Method: GET 2024-04-09T22:02:42.216 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.216 DEBUG:boto:Data: 2024-04-09T22:02:42.216 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.216 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.216 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.216 DEBUG:boto:Params: {} 2024-04-09T22:02:42.217 DEBUG:boto:Token: None 2024-04-09T22:02:42.217 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.217 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffbf77a767972562-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.219 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.220 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.220 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.220 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.220 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.220 DEBUG:boto:Method: GET 2024-04-09T22:02:42.220 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.220 DEBUG:boto:Data: 2024-04-09T22:02:42.220 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.221 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.221 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.221 DEBUG:boto:Params: {} 2024-04-09T22:02:42.221 DEBUG:boto:Token: None 2024-04-09T22:02:42.221 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.221 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053aed4bca767506b-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.224 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.224 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.224 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.224 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.224 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.224 DEBUG:boto:Data: 2024-04-09T22:02:42.224 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.225 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.225 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.225 DEBUG:boto:Params: {} 2024-04-09T22:02:42.225 DEBUG:boto:Token: None 2024-04-09T22:02:42.225 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.225 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.228 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009652c5bccf1f455e-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.228 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.228 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.228 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.229 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.229 DEBUG:boto:Data: 2024-04-09T22:02:42.229 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.229 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.229 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.229 DEBUG:boto:Params: {} 2024-04-09T22:02:42.229 DEBUG:boto:Token: None 2024-04-09T22:02:42.229 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.229 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.232 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000266aec01d1ada7a2-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.233 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.233 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.233 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.233 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.233 DEBUG:boto:Method: GET 2024-04-09T22:02:42.233 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.233 DEBUG:boto:Data: 2024-04-09T22:02:42.233 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.233 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.233 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.234 DEBUG:boto:Params: {} 2024-04-09T22:02:42.234 DEBUG:boto:Token: None 2024-04-09T22:02:42.234 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.234 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.236 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6e3062f69cdd732-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.278 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.279 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.279 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.279 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.279 DEBUG:boto:Method: GET 2024-04-09T22:02:42.279 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.279 DEBUG:boto:Data: 2024-04-09T22:02:42.279 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.279 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.279 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.279 DEBUG:boto:Params: {} 2024-04-09T22:02:42.279 DEBUG:boto:Token: None 2024-04-09T22:02:42.280 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.280 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d5e657aab375445-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.282 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.282 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.282 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.283 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.283 DEBUG:boto:Method: GET 2024-04-09T22:02:42.283 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.283 DEBUG:boto:Data: 2024-04-09T22:02:42.283 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.283 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.283 DEBUG:boto:Params: {} 2024-04-09T22:02:42.283 DEBUG:boto:Token: None 2024-04-09T22:02:42.283 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.283 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464fdf49325a34d1-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.286 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.287 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.287 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.287 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.287 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.287 DEBUG:boto:Method: GET 2024-04-09T22:02:42.287 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.287 DEBUG:boto:Data: 2024-04-09T22:02:42.287 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.287 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.288 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.288 DEBUG:boto:Params: {} 2024-04-09T22:02:42.288 DEBUG:boto:Token: None 2024-04-09T22:02:42.288 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.288 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f776531114cab45-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.291 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.291 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.292 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.292 DEBUG:boto:path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.292 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.292 DEBUG:boto:Method: GET 2024-04-09T22:02:42.292 DEBUG:boto:Path: /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.292 DEBUG:boto:Data: 2024-04-09T22:02:42.292 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.292 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.292 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.292 DEBUG:boto:Params: {} 2024-04-09T22:02:42.292 DEBUG:boto:Token: None 2024-04-09T22:02:42.293 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.293 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew= 2024-04-09T22:02:42.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew=', 'Content-Length': '0'} 2024-04-09T22:02:42.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062898dff5f6e3606-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.295 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.295 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.296 DEBUG:boto:path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.296 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.296 DEBUG:boto:Method: GET 2024-04-09T22:02:42.296 DEBUG:boto:Path: /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.296 DEBUG:boto:Data: 2024-04-09T22:02:42.296 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.296 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.296 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.296 DEBUG:boto:Params: {} 2024-04-09T22:02:42.296 DEBUG:boto:Token: None 2024-04-09T22:02:42.297 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.297 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew= 2024-04-09T22:02:42.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew=', 'Content-Length': '0'} 2024-04-09T22:02:42.299 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d49a3aa2e099c6ae-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.342 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.343 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.343 DEBUG:boto:path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.343 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.343 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.344 DEBUG:boto:Path: /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.344 DEBUG:boto:Data: 2024-04-09T22:02:42.344 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.344 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.344 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.344 DEBUG:boto:Params: {} 2024-04-09T22:02:42.345 DEBUG:boto:Token: None 2024-04-09T22:02:42.345 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.345 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QV9UvJ+XxVwT/u8HY3DJEPOey0A= 2024-04-09T22:02:42.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QV9UvJ+XxVwT/u8HY3DJEPOey0A=', 'Content-Length': '0'} 2024-04-09T22:02:42.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000606d7b6965ce1280-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.348 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.348 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.348 DEBUG:boto:path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.349 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.349 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.349 DEBUG:boto:Path: /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.349 DEBUG:boto:Data: 2024-04-09T22:02:42.349 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.349 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.349 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.350 DEBUG:boto:Params: {} 2024-04-09T22:02:42.350 DEBUG:boto:Token: None 2024-04-09T22:02:42.350 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.350 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QV9UvJ+XxVwT/u8HY3DJEPOey0A= 2024-04-09T22:02:42.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QV9UvJ+XxVwT/u8HY3DJEPOey0A=', 'Content-Length': '0'} 2024-04-09T22:02:42.352 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000772dea47f22a534d-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.353 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.353 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.353 DEBUG:boto:path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.353 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.353 DEBUG:boto:Method: GET 2024-04-09T22:02:42.353 DEBUG:boto:Path: /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.353 DEBUG:boto:Data: 2024-04-09T22:02:42.353 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.353 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.353 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.353 DEBUG:boto:Params: {} 2024-04-09T22:02:42.353 DEBUG:boto:Token: None 2024-04-09T22:02:42.354 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.354 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew= 2024-04-09T22:02:42.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew=', 'Content-Length': '0'} 2024-04-09T22:02:42.356 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027b1e0ceff7594d9-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.398 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.399 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.399 DEBUG:boto:path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.399 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.399 DEBUG:boto:Method: GET 2024-04-09T22:02:42.399 DEBUG:boto:Path: /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.399 DEBUG:boto:Data: 2024-04-09T22:02:42.399 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.399 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.399 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.399 DEBUG:boto:Params: {} 2024-04-09T22:02:42.400 DEBUG:boto:Token: None 2024-04-09T22:02:42.400 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E 2024-04-09T22:02:42.400 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew= 2024-04-09T22:02:42.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:etSduDFSBbXURcwIzauTI/QDLew=', 'Content-Length': '0'} 2024-04-09T22:02:42.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'J3pYxHY6NfaDDMPsw0PjqBhuWJT5n5E'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004227f55f921e168f-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.402 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.402 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.403 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.403 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.403 DEBUG:boto:Method: GET 2024-04-09T22:02:42.403 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.403 DEBUG:boto:Data: 2024-04-09T22:02:42.403 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.403 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.403 DEBUG:boto:Params: {} 2024-04-09T22:02:42.403 DEBUG:boto:Token: None 2024-04-09T22:02:42.404 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035050885e6d1c26a-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.450 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.451 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.451 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.451 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.451 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.452 DEBUG:boto:Method: GET 2024-04-09T22:02:42.452 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.452 DEBUG:boto:Data: 2024-04-09T22:02:42.452 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.452 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.452 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.452 DEBUG:boto:Params: {} 2024-04-09T22:02:42.452 DEBUG:boto:Token: None 2024-04-09T22:02:42.452 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.453 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc5ead2b85feb331-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.455 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.456 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.456 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.456 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.456 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.456 DEBUG:boto:Data: 2024-04-09T22:02:42.456 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.456 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.457 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.457 DEBUG:boto:Params: {} 2024-04-09T22:02:42.457 DEBUG:boto:Token: None 2024-04-09T22:02:42.457 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.457 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e29a669f8329032d-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.461 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.461 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.461 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.462 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.462 DEBUG:boto:Data: 2024-04-09T22:02:42.462 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.462 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.462 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.462 DEBUG:boto:Params: {} 2024-04-09T22:02:42.462 DEBUG:boto:Token: None 2024-04-09T22:02:42.462 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.462 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.466 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000038efe867b94723c3-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.466 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.466 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.466 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.467 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.467 DEBUG:boto:Method: GET 2024-04-09T22:02:42.467 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.467 DEBUG:boto:Data: 2024-04-09T22:02:42.467 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.467 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.467 DEBUG:boto:Params: {} 2024-04-09T22:02:42.467 DEBUG:boto:Token: None 2024-04-09T22:02:42.467 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.469 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f02fb605d7f3783d-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.510 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.511 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.511 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.511 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.511 DEBUG:boto:Method: GET 2024-04-09T22:02:42.511 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.511 DEBUG:boto:Data: 2024-04-09T22:02:42.511 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.511 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.511 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.511 DEBUG:boto:Params: {} 2024-04-09T22:02:42.512 DEBUG:boto:Token: None 2024-04-09T22:02:42.512 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.512 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.514 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000836f3bc1e9bd9cca-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.514 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.514 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.515 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.515 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.515 DEBUG:boto:Method: GET 2024-04-09T22:02:42.515 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.515 DEBUG:boto:Data: 2024-04-09T22:02:42.515 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.515 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.515 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.515 DEBUG:boto:Params: {} 2024-04-09T22:02:42.515 DEBUG:boto:Token: None 2024-04-09T22:02:42.515 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.516 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079c5c8b168e09cf5-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.558 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.559 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.559 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.559 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.559 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.559 DEBUG:boto:Method: GET 2024-04-09T22:02:42.559 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.559 DEBUG:boto:Data: 2024-04-09T22:02:42.560 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.560 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.560 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.560 DEBUG:boto:Params: {} 2024-04-09T22:02:42.560 DEBUG:boto:Token: None 2024-04-09T22:02:42.560 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.560 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018ffb501138a3094-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.563 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.564 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.564 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.564 DEBUG:boto:path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.564 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.564 DEBUG:boto:Method: GET 2024-04-09T22:02:42.564 DEBUG:boto:Path: /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.564 DEBUG:boto:Data: 2024-04-09T22:02:42.564 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.564 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.564 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.564 DEBUG:boto:Params: {} 2024-04-09T22:02:42.565 DEBUG:boto:Token: None 2024-04-09T22:02:42.565 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.565 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA= 2024-04-09T22:02:42.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA=', 'Content-Length': '0'} 2024-04-09T22:02:42.567 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c71f6c12a3661195-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.567 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.567 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.567 DEBUG:boto:path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.567 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.567 DEBUG:boto:Method: GET 2024-04-09T22:02:42.567 DEBUG:boto:Path: /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.568 DEBUG:boto:Data: 2024-04-09T22:02:42.568 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.568 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.568 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.568 DEBUG:boto:Params: {} 2024-04-09T22:02:42.568 DEBUG:boto:Token: None 2024-04-09T22:02:42.568 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.568 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA= 2024-04-09T22:02:42.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA=', 'Content-Length': '0'} 2024-04-09T22:02:42.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062853fc0aa259c71-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.618 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.619 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.619 DEBUG:boto:path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.619 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.619 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.619 DEBUG:boto:Path: /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.619 DEBUG:boto:Data: 2024-04-09T22:02:42.619 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.619 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.619 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.619 DEBUG:boto:Params: {} 2024-04-09T22:02:42.620 DEBUG:boto:Token: None 2024-04-09T22:02:42.620 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.620 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4zGc7catXsp4bir0UxhnJAAcd/c= 2024-04-09T22:02:42.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4zGc7catXsp4bir0UxhnJAAcd/c=', 'Content-Length': '0'} 2024-04-09T22:02:42.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ecd9045a83f956bd-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.622 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.622 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.622 DEBUG:boto:path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.622 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.623 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.623 DEBUG:boto:Path: /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.623 DEBUG:boto:Data: 2024-04-09T22:02:42.623 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.623 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.623 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.623 DEBUG:boto:Params: {} 2024-04-09T22:02:42.623 DEBUG:boto:Token: None 2024-04-09T22:02:42.623 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.623 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4zGc7catXsp4bir0UxhnJAAcd/c= 2024-04-09T22:02:42.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4zGc7catXsp4bir0UxhnJAAcd/c=', 'Content-Length': '0'} 2024-04-09T22:02:42.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d3a8a218f9b50fd2-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.626 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.626 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.626 DEBUG:boto:path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.626 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.626 DEBUG:boto:Method: GET 2024-04-09T22:02:42.626 DEBUG:boto:Path: /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.626 DEBUG:boto:Data: 2024-04-09T22:02:42.627 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.627 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.627 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.627 DEBUG:boto:Params: {} 2024-04-09T22:02:42.627 DEBUG:boto:Token: None 2024-04-09T22:02:42.627 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.627 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA= 2024-04-09T22:02:42.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA=', 'Content-Length': '0'} 2024-04-09T22:02:42.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb97ba975e5a43e7-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.629 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.629 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.629 DEBUG:boto:path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.630 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.630 DEBUG:boto:Method: GET 2024-04-09T22:02:42.630 DEBUG:boto:Path: /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.630 DEBUG:boto:Data: 2024-04-09T22:02:42.630 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.630 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.630 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.630 DEBUG:boto:Params: {} 2024-04-09T22:02:42.630 DEBUG:boto:Token: None 2024-04-09T22:02:42.630 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8 2024-04-09T22:02:42.631 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA= 2024-04-09T22:02:42.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PPH0vCb8M8AyOGiU6TF8vLnZrpA=', 'Content-Length': '0'} 2024-04-09T22:02:42.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'nR0R8ccnFLtMtmKXAhGHxsgDs8klFV8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c4db99df473e05a-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.633 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.633 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.633 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.633 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.633 DEBUG:boto:Method: GET 2024-04-09T22:02:42.633 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.633 DEBUG:boto:Data: 2024-04-09T22:02:42.633 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.633 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.634 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.634 DEBUG:boto:Params: {} 2024-04-09T22:02:42.634 DEBUG:boto:Token: None 2024-04-09T22:02:42.634 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.634 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf32f8bf7a094ae0-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.637 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.637 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.637 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.637 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.638 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.638 DEBUG:boto:Method: GET 2024-04-09T22:02:42.638 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.638 DEBUG:boto:Data: 2024-04-09T22:02:42.638 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.638 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.638 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.638 DEBUG:boto:Params: {} 2024-04-09T22:02:42.638 DEBUG:boto:Token: None 2024-04-09T22:02:42.638 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.639 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d45ad567dd8b45c-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.641 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.642 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.642 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.642 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.642 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.642 DEBUG:boto:Data: 2024-04-09T22:02:42.642 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.642 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.642 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.643 DEBUG:boto:Params: {} 2024-04-09T22:02:42.643 DEBUG:boto:Token: None 2024-04-09T22:02:42.643 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.643 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000083092c7592694c6a-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.646 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.646 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.646 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.646 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.647 DEBUG:boto:Data: 2024-04-09T22:02:42.647 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.647 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.647 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.647 DEBUG:boto:Params: {} 2024-04-09T22:02:42.647 DEBUG:boto:Token: None 2024-04-09T22:02:42.647 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.647 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f3729db205f88bf-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.651 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.651 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.651 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.651 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.651 DEBUG:boto:Method: GET 2024-04-09T22:02:42.651 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.651 DEBUG:boto:Data: 2024-04-09T22:02:42.651 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.651 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.652 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.652 DEBUG:boto:Params: {} 2024-04-09T22:02:42.652 DEBUG:boto:Token: None 2024-04-09T22:02:42.652 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.652 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.654 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094877072bdb3be23-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.694 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.695 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.695 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.695 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.695 DEBUG:boto:Method: GET 2024-04-09T22:02:42.695 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.695 DEBUG:boto:Data: 2024-04-09T22:02:42.695 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.695 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.695 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.695 DEBUG:boto:Params: {} 2024-04-09T22:02:42.696 DEBUG:boto:Token: None 2024-04-09T22:02:42.696 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.696 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.698 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007ba4e2309d6a3c3c-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.698 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.698 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.698 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.699 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.699 DEBUG:boto:Method: GET 2024-04-09T22:02:42.699 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.699 DEBUG:boto:Data: 2024-04-09T22:02:42.699 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.699 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.699 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.699 DEBUG:boto:Params: {} 2024-04-09T22:02:42.699 DEBUG:boto:Token: None 2024-04-09T22:02:42.699 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.700 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ffb8adb16c4e8d1-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.702 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.703 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.703 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.703 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.703 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.703 DEBUG:boto:Method: GET 2024-04-09T22:02:42.703 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.703 DEBUG:boto:Data: 2024-04-09T22:02:42.703 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.703 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.703 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.703 DEBUG:boto:Params: {} 2024-04-09T22:02:42.703 DEBUG:boto:Token: None 2024-04-09T22:02:42.704 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.704 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e54801201f8a926-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.706 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.707 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.707 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.707 DEBUG:boto:path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.707 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.707 DEBUG:boto:Method: GET 2024-04-09T22:02:42.707 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.708 DEBUG:boto:Data: 2024-04-09T22:02:42.708 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.708 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.708 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.708 DEBUG:boto:Params: {} 2024-04-09T22:02:42.708 DEBUG:boto:Token: None 2024-04-09T22:02:42.708 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.708 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs= 2024-04-09T22:02:42.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs=', 'Content-Length': '0'} 2024-04-09T22:02:42.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000242da926d2c24108-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.711 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.711 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.711 DEBUG:boto:path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.711 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.711 DEBUG:boto:Method: GET 2024-04-09T22:02:42.711 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.711 DEBUG:boto:Data: 2024-04-09T22:02:42.711 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.711 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.711 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.712 DEBUG:boto:Params: {} 2024-04-09T22:02:42.712 DEBUG:boto:Token: None 2024-04-09T22:02:42.712 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.712 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs= 2024-04-09T22:02:42.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs=', 'Content-Length': '0'} 2024-04-09T22:02:42.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000094cc6a799d9557ca-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.714 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.714 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.714 DEBUG:boto:path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.714 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.714 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.715 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.715 DEBUG:boto:Data: 2024-04-09T22:02:42.715 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.715 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.715 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.715 DEBUG:boto:Params: {} 2024-04-09T22:02:42.715 DEBUG:boto:Token: None 2024-04-09T22:02:42.715 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.715 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XbbYwhSIQ4q5C+80A1vZKUfS7qk= 2024-04-09T22:02:42.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XbbYwhSIQ4q5C+80A1vZKUfS7qk=', 'Content-Length': '0'} 2024-04-09T22:02:42.717 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa377bc6986316aa-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.717 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.717 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.718 DEBUG:boto:path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.718 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.718 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.718 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.718 DEBUG:boto:Data: 2024-04-09T22:02:42.718 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.718 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.718 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.718 DEBUG:boto:Params: {} 2024-04-09T22:02:42.718 DEBUG:boto:Token: None 2024-04-09T22:02:42.719 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.719 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XbbYwhSIQ4q5C+80A1vZKUfS7qk= 2024-04-09T22:02:42.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XbbYwhSIQ4q5C+80A1vZKUfS7qk=', 'Content-Length': '0'} 2024-04-09T22:02:42.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058122a1740cb923a-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.721 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.721 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.721 DEBUG:boto:path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.721 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.721 DEBUG:boto:Method: GET 2024-04-09T22:02:42.721 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.721 DEBUG:boto:Data: 2024-04-09T22:02:42.721 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.721 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.721 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.722 DEBUG:boto:Params: {} 2024-04-09T22:02:42.722 DEBUG:boto:Token: None 2024-04-09T22:02:42.722 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.722 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs= 2024-04-09T22:02:42.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs=', 'Content-Length': '0'} 2024-04-09T22:02:42.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061782ffbad4d3d6b-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.766 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.767 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.767 DEBUG:boto:path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.767 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.767 DEBUG:boto:Method: GET 2024-04-09T22:02:42.767 DEBUG:boto:Path: /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.767 DEBUG:boto:Data: 2024-04-09T22:02:42.767 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.767 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.767 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.767 DEBUG:boto:Params: {} 2024-04-09T22:02:42.767 DEBUG:boto:Token: None 2024-04-09T22:02:42.768 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn 2024-04-09T22:02:42.768 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs= 2024-04-09T22:02:42.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XSXD71eeMaWshRWPGVqAMG5knGs=', 'Content-Length': '0'} 2024-04-09T22:02:42.770 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '9.828cYFy0Yv6yg.oVxTKG-.bzjWzyn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000212fc42d8932a2fd-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.770 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.770 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.770 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.770 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.770 DEBUG:boto:Method: GET 2024-04-09T22:02:42.771 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.771 DEBUG:boto:Data: 2024-04-09T22:02:42.771 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.771 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.771 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.771 DEBUG:boto:Params: {} 2024-04-09T22:02:42.771 DEBUG:boto:Token: None 2024-04-09T22:02:42.771 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.771 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b09ef736e6775d41-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.774 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.775 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.775 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.775 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.775 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.775 DEBUG:boto:Method: GET 2024-04-09T22:02:42.775 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.775 DEBUG:boto:Data: 2024-04-09T22:02:42.775 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.775 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.775 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.776 DEBUG:boto:Params: {} 2024-04-09T22:02:42.776 DEBUG:boto:Token: None 2024-04-09T22:02:42.776 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.776 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000070b58a7d7a59fca-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.779 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.779 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.779 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.779 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.779 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.779 DEBUG:boto:Data: 2024-04-09T22:02:42.780 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.780 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.780 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.780 DEBUG:boto:Params: {} 2024-04-09T22:02:42.780 DEBUG:boto:Token: None 2024-04-09T22:02:42.780 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.780 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.783 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a18ed888cc241d8-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.783 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.784 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.784 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.784 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:42.784 DEBUG:boto:Data: 2024-04-09T22:02:42.784 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.784 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.784 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.784 DEBUG:boto:Params: {} 2024-04-09T22:02:42.784 DEBUG:boto:Token: None 2024-04-09T22:02:42.784 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj 2024-04-09T22:02:42.785 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM= 2024-04-09T22:02:42.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dMFET4Esap036Br5bSyBc2RBysM=', 'Content-Length': '0'} 2024-04-09T22:02:42.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a888b18c2a60b087-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.788 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.788 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.788 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.788 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.789 DEBUG:boto:Method: GET 2024-04-09T22:02:42.789 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.789 DEBUG:boto:Data: 2024-04-09T22:02:42.789 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.789 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.789 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.789 DEBUG:boto:Params: {} 2024-04-09T22:02:42.789 DEBUG:boto:Token: None 2024-04-09T22:02:42.789 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.789 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000037a96cb6c99bce4a-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.834 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.835 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.835 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.835 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.835 DEBUG:boto:Method: GET 2024-04-09T22:02:42.835 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.835 DEBUG:boto:Data: 2024-04-09T22:02:42.835 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.835 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.835 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.835 DEBUG:boto:Params: {} 2024-04-09T22:02:42.836 DEBUG:boto:Token: None 2024-04-09T22:02:42.836 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:42.836 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE= 2024-04-09T22:02:42.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JcxX4ZcIVIR4Iloy2+69HOIRDLE=', 'Content-Length': '0'} 2024-04-09T22:02:42.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067c4a988a19b8da3-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.838 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.838 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.838 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.839 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.839 DEBUG:boto:Method: GET 2024-04-09T22:02:42.839 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.839 DEBUG:boto:Data: 2024-04-09T22:02:42.839 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.839 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.839 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.839 DEBUG:boto:Params: {} 2024-04-09T22:02:42.839 DEBUG:boto:Token: None 2024-04-09T22:02:42.839 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.840 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e470027759a9236-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.886 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.887 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.887 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.887 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.887 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.887 DEBUG:boto:Method: GET 2024-04-09T22:02:42.887 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.887 DEBUG:boto:Data: 2024-04-09T22:02:42.887 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.888 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.888 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.888 DEBUG:boto:Params: {} 2024-04-09T22:02:42.888 DEBUG:boto:Token: None 2024-04-09T22:02:42.888 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.888 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6aae118de97f6d9-006615bb02-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.891 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:42.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.892 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.892 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.892 DEBUG:boto:path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.892 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.892 DEBUG:boto:Method: GET 2024-04-09T22:02:42.892 DEBUG:boto:Path: /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.892 DEBUG:boto:Data: 2024-04-09T22:02:42.892 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.892 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.892 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.892 DEBUG:boto:Params: {} 2024-04-09T22:02:42.893 DEBUG:boto:Token: None 2024-04-09T22:02:42.893 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.893 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0= 2024-04-09T22:02:42.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0=', 'Content-Length': '0'} 2024-04-09T22:02:42.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000700c66aabc20a7f5-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.895 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.895 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.895 DEBUG:boto:path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.895 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.896 DEBUG:boto:Method: GET 2024-04-09T22:02:42.896 DEBUG:boto:Path: /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.896 DEBUG:boto:Data: 2024-04-09T22:02:42.896 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.896 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.896 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.896 DEBUG:boto:Params: {} 2024-04-09T22:02:42.896 DEBUG:boto:Token: None 2024-04-09T22:02:42.896 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.896 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0= 2024-04-09T22:02:42.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0=', 'Content-Length': '0'} 2024-04-09T22:02:42.898 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5259bf73bb1d51b-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.898 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.899 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.899 DEBUG:boto:path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.899 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.899 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.899 DEBUG:boto:Path: /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.899 DEBUG:boto:Data: 2024-04-09T22:02:42.899 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.899 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.899 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.899 DEBUG:boto:Params: {} 2024-04-09T22:02:42.899 DEBUG:boto:Token: None 2024-04-09T22:02:42.900 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.900 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sBzn8clz9jsDb8Zn1gJWRBrmBvQ= 2024-04-09T22:02:42.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sBzn8clz9jsDb8Zn1gJWRBrmBvQ=', 'Content-Length': '0'} 2024-04-09T22:02:42.902 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009552a7def5ce082b-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.902 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.902 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.902 DEBUG:boto:path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.902 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.902 DEBUG:boto:Method: HEAD 2024-04-09T22:02:42.902 DEBUG:boto:Path: /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.902 DEBUG:boto:Data: 2024-04-09T22:02:42.903 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.903 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.903 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.903 DEBUG:boto:Params: {} 2024-04-09T22:02:42.903 DEBUG:boto:Token: None 2024-04-09T22:02:42.903 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.903 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sBzn8clz9jsDb8Zn1gJWRBrmBvQ= 2024-04-09T22:02:42.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sBzn8clz9jsDb8Zn1gJWRBrmBvQ=', 'Content-Length': '0'} 2024-04-09T22:02:42.905 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000097e1352ca551cd57-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:42.905 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.906 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.906 DEBUG:boto:path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.906 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.906 DEBUG:boto:Method: GET 2024-04-09T22:02:42.906 DEBUG:boto:Path: /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.906 DEBUG:boto:Data: 2024-04-09T22:02:42.906 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.906 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.906 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.906 DEBUG:boto:Params: {} 2024-04-09T22:02:42.906 DEBUG:boto:Token: None 2024-04-09T22:02:42.907 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.907 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0= 2024-04-09T22:02:42.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0=', 'Content-Length': '0'} 2024-04-09T22:02:42.909 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068b914c7fb240d79-006615bb02-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.954 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:42.955 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:42.955 DEBUG:boto:path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.955 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.956 DEBUG:boto:Method: GET 2024-04-09T22:02:42.956 DEBUG:boto:Path: /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.956 DEBUG:boto:Data: 2024-04-09T22:02:42.956 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.956 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:42.956 DEBUG:boto:Port: 8000 2024-04-09T22:02:42.956 DEBUG:boto:Params: {} 2024-04-09T22:02:42.956 DEBUG:boto:Token: None 2024-04-09T22:02:42.956 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/obj?versionId=NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD 2024-04-09T22:02:42.956 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0= 2024-04-09T22:02:42.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XUdzmQLPnAF/J+PDQcQ2X53iTU0=', 'Content-Length': '0'} 2024-04-09T22:02:42.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'NGL-GxJyNdNy67KIPsaQVKx4ZMNZGZD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000527bd4f235c4bf27-006615bb02-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:42.959 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:42.959 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:42.959 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.959 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:42.959 DEBUG:boto:Method: GET 2024-04-09T22:02:42.959 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:42.960 DEBUG:boto:Data: 2024-04-09T22:02:42.960 DEBUG:boto:Headers: {} 2024-04-09T22:02:42.960 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:42.960 DEBUG:boto:Port: 8001 2024-04-09T22:02:42.960 DEBUG:boto:Params: {} 2024-04-09T22:02:42.960 DEBUG:boto:Token: None 2024-04-09T22:02:42.960 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:42 GMT /xryqqk-15/ 2024-04-09T22:02:42.960 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c= 2024-04-09T22:02:42.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:U+Wozg1SF2IpylEbowFeUFZ/z6c=', 'Content-Length': '0'} 2024-04-09T22:02:42.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db50a2a8288eafa8-006615bb02-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.010 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.011 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.011 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.011 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.011 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.011 DEBUG:boto:Method: GET 2024-04-09T22:02:43.011 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.012 DEBUG:boto:Data: 2024-04-09T22:02:43.012 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.012 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.012 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.012 DEBUG:boto:Params: {} 2024-04-09T22:02:43.012 DEBUG:boto:Token: None 2024-04-09T22:02:43.012 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.012 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc2b623917eddb3-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.015 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.015 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.016 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.016 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.016 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.016 DEBUG:boto:Data: 2024-04-09T22:02:43.016 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.016 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.016 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.016 DEBUG:boto:Params: {} 2024-04-09T22:02:43.016 DEBUG:boto:Token: None 2024-04-09T22:02:43.016 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.017 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.020 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000004062f10541840-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.020 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.020 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.020 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.020 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.021 DEBUG:boto:Data: 2024-04-09T22:02:43.021 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.021 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.021 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.021 DEBUG:boto:Params: {} 2024-04-09T22:02:43.021 DEBUG:boto:Token: None 2024-04-09T22:02:43.021 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.021 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009bac1a3a912defa1-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.025 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.025 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.025 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.025 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.025 DEBUG:boto:Method: GET 2024-04-09T22:02:43.025 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.025 DEBUG:boto:Data: 2024-04-09T22:02:43.025 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.025 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.025 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.026 DEBUG:boto:Params: {} 2024-04-09T22:02:43.026 DEBUG:boto:Token: None 2024-04-09T22:02:43.026 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.026 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.028 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f8632076866074e-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.070 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.071 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.071 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.071 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.071 DEBUG:boto:Method: GET 2024-04-09T22:02:43.071 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.071 DEBUG:boto:Data: 2024-04-09T22:02:43.071 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.071 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.071 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.071 DEBUG:boto:Params: {} 2024-04-09T22:02:43.071 DEBUG:boto:Token: None 2024-04-09T22:02:43.072 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.072 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.074 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070f8945e5bc8faf0-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.074 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.074 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.075 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.075 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.075 DEBUG:boto:Method: GET 2024-04-09T22:02:43.075 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.075 DEBUG:boto:Data: 2024-04-09T22:02:43.075 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.075 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.075 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.075 DEBUG:boto:Params: {} 2024-04-09T22:02:43.075 DEBUG:boto:Token: None 2024-04-09T22:02:43.076 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.076 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063dd931b4055057b-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.126 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.127 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.127 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.127 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.127 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.127 DEBUG:boto:Method: GET 2024-04-09T22:02:43.127 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.127 DEBUG:boto:Data: 2024-04-09T22:02:43.127 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.128 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.128 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.128 DEBUG:boto:Params: {} 2024-04-09T22:02:43.128 DEBUG:boto:Token: None 2024-04-09T22:02:43.128 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.128 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000305aca2cefb42ce7-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.131 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.132 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.132 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.132 DEBUG:boto:path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.132 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.132 DEBUG:boto:Method: GET 2024-04-09T22:02:43.132 DEBUG:boto:Path: /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.132 DEBUG:boto:Data: 2024-04-09T22:02:43.132 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.132 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.132 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.133 DEBUG:boto:Params: {} 2024-04-09T22:02:43.133 DEBUG:boto:Token: None 2024-04-09T22:02:43.133 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.133 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw= 2024-04-09T22:02:43.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw=', 'Content-Length': '0'} 2024-04-09T22:02:43.136 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e2f30ac6f8c8ad73-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.136 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.136 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.136 DEBUG:boto:path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.136 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.136 DEBUG:boto:Method: GET 2024-04-09T22:02:43.136 DEBUG:boto:Path: /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.137 DEBUG:boto:Data: 2024-04-09T22:02:43.137 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.137 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.137 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.137 DEBUG:boto:Params: {} 2024-04-09T22:02:43.137 DEBUG:boto:Token: None 2024-04-09T22:02:43.137 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.137 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw= 2024-04-09T22:02:43.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw=', 'Content-Length': '0'} 2024-04-09T22:02:43.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b66b42359df90b3-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.139 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.140 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.140 DEBUG:boto:path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.140 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.140 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.140 DEBUG:boto:Path: /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.140 DEBUG:boto:Data: 2024-04-09T22:02:43.140 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.140 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.140 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.140 DEBUG:boto:Params: {} 2024-04-09T22:02:43.141 DEBUG:boto:Token: None 2024-04-09T22:02:43.141 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.141 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yBf0RhubrAU0ZbBB19VXl5GQkEY= 2024-04-09T22:02:43.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yBf0RhubrAU0ZbBB19VXl5GQkEY=', 'Content-Length': '0'} 2024-04-09T22:02:43.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6fde14a8fafac0c-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.143 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.143 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.143 DEBUG:boto:path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.143 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.143 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.143 DEBUG:boto:Path: /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.143 DEBUG:boto:Data: 2024-04-09T22:02:43.144 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.144 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.144 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.144 DEBUG:boto:Params: {} 2024-04-09T22:02:43.144 DEBUG:boto:Token: None 2024-04-09T22:02:43.144 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yBf0RhubrAU0ZbBB19VXl5GQkEY= 2024-04-09T22:02:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yBf0RhubrAU0ZbBB19VXl5GQkEY=', 'Content-Length': '0'} 2024-04-09T22:02:43.146 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008087b70f19e9e6b7-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.146 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.146 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.147 DEBUG:boto:path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.147 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.147 DEBUG:boto:Method: GET 2024-04-09T22:02:43.147 DEBUG:boto:Path: /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.147 DEBUG:boto:Data: 2024-04-09T22:02:43.147 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.147 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.147 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.147 DEBUG:boto:Params: {} 2024-04-09T22:02:43.147 DEBUG:boto:Token: None 2024-04-09T22:02:43.147 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.148 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw= 2024-04-09T22:02:43.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw=', 'Content-Length': '0'} 2024-04-09T22:02:43.149 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1caec2d3b8757ce-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.194 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.195 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.195 DEBUG:boto:path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.195 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.195 DEBUG:boto:Method: GET 2024-04-09T22:02:43.195 DEBUG:boto:Path: /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.195 DEBUG:boto:Data: 2024-04-09T22:02:43.195 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.195 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.195 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.195 DEBUG:boto:Params: {} 2024-04-09T22:02:43.195 DEBUG:boto:Token: None 2024-04-09T22:02:43.196 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=.rWxN.f1EH02C.PAOjorQnSaCM.GZJP 2024-04-09T22:02:43.196 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw= 2024-04-09T22:02:43.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MdcAJeTxOpidyZmcE+TbZHjPMrw=', 'Content-Length': '0'} 2024-04-09T22:02:43.198 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '.rWxN.f1EH02C.PAOjorQnSaCM.GZJP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003baeed0fc3e07724-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.199 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.199 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.199 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.199 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.199 DEBUG:boto:Method: GET 2024-04-09T22:02:43.199 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.199 DEBUG:boto:Data: 2024-04-09T22:02:43.199 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.199 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.199 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.199 DEBUG:boto:Params: {} 2024-04-09T22:02:43.200 DEBUG:boto:Token: None 2024-04-09T22:02:43.200 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.200 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3f19b9bc67ae6ec-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.202 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.203 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.203 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.203 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.205 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.205 DEBUG:boto:Method: GET 2024-04-09T22:02:43.205 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.205 DEBUG:boto:Data: 2024-04-09T22:02:43.206 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.206 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.206 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.206 DEBUG:boto:Params: {} 2024-04-09T22:02:43.206 DEBUG:boto:Token: None 2024-04-09T22:02:43.206 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.206 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d469d0460c2d709a-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.209 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.209 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.209 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.210 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.210 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.210 DEBUG:boto:Data: 2024-04-09T22:02:43.210 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.210 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.210 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.210 DEBUG:boto:Params: {} 2024-04-09T22:02:43.210 DEBUG:boto:Token: None 2024-04-09T22:02:43.210 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.211 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.214 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000529c413fd68ff7d5-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.214 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.214 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.214 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.214 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.214 DEBUG:boto:Data: 2024-04-09T22:02:43.214 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.214 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.215 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.215 DEBUG:boto:Params: {} 2024-04-09T22:02:43.215 DEBUG:boto:Token: None 2024-04-09T22:02:43.215 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.215 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.218 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a2835cea32874d86-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.218 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.218 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.219 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.219 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.219 DEBUG:boto:Method: GET 2024-04-09T22:02:43.219 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.219 DEBUG:boto:Data: 2024-04-09T22:02:43.219 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.219 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.219 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.219 DEBUG:boto:Params: {} 2024-04-09T22:02:43.219 DEBUG:boto:Token: None 2024-04-09T22:02:43.220 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.220 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.221 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004ad8e7aaa1408a27-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.262 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.263 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.263 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.263 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.263 DEBUG:boto:Method: GET 2024-04-09T22:02:43.263 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.263 DEBUG:boto:Data: 2024-04-09T22:02:43.263 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.263 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.263 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.263 DEBUG:boto:Params: {} 2024-04-09T22:02:43.263 DEBUG:boto:Token: None 2024-04-09T22:02:43.264 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.264 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.266 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e931bcffdc1b6355-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.266 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.266 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.266 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.266 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.266 DEBUG:boto:Method: GET 2024-04-09T22:02:43.266 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.267 DEBUG:boto:Data: 2024-04-09T22:02:43.267 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.267 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.267 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.267 DEBUG:boto:Params: {} 2024-04-09T22:02:43.267 DEBUG:boto:Token: None 2024-04-09T22:02:43.267 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.267 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e50dac9835496905-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.310 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.311 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.311 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.311 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.311 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.311 DEBUG:boto:Method: GET 2024-04-09T22:02:43.311 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.311 DEBUG:boto:Data: 2024-04-09T22:02:43.311 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.312 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.312 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.312 DEBUG:boto:Params: {} 2024-04-09T22:02:43.312 DEBUG:boto:Token: None 2024-04-09T22:02:43.312 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.312 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e29d719e1667b5-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.315 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.315 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.316 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.316 DEBUG:boto:path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.316 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.316 DEBUG:boto:Method: GET 2024-04-09T22:02:43.316 DEBUG:boto:Path: /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.316 DEBUG:boto:Data: 2024-04-09T22:02:43.316 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.316 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.316 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.316 DEBUG:boto:Params: {} 2024-04-09T22:02:43.316 DEBUG:boto:Token: None 2024-04-09T22:02:43.317 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.317 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA= 2024-04-09T22:02:43.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA=', 'Content-Length': '0'} 2024-04-09T22:02:43.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd5fed4474280673-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.319 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.319 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.319 DEBUG:boto:path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.319 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.319 DEBUG:boto:Method: GET 2024-04-09T22:02:43.319 DEBUG:boto:Path: /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.319 DEBUG:boto:Data: 2024-04-09T22:02:43.319 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.319 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.320 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.320 DEBUG:boto:Params: {} 2024-04-09T22:02:43.320 DEBUG:boto:Token: None 2024-04-09T22:02:43.320 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA= 2024-04-09T22:02:43.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA=', 'Content-Length': '0'} 2024-04-09T22:02:43.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff65a107d4cb7c6b-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.362 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.363 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.363 DEBUG:boto:path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.363 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.363 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.363 DEBUG:boto:Path: /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.363 DEBUG:boto:Data: 2024-04-09T22:02:43.363 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.363 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.363 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.363 DEBUG:boto:Params: {} 2024-04-09T22:02:43.364 DEBUG:boto:Token: None 2024-04-09T22:02:43.364 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.364 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7Y0sHncTxmBZeaiVg85X+LrgV+Y= 2024-04-09T22:02:43.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Y0sHncTxmBZeaiVg85X+LrgV+Y=', 'Content-Length': '0'} 2024-04-09T22:02:43.366 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d42482b281728930-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.366 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.366 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.366 DEBUG:boto:path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.366 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.367 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.367 DEBUG:boto:Path: /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.367 DEBUG:boto:Data: 2024-04-09T22:02:43.367 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.367 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.367 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.367 DEBUG:boto:Params: {} 2024-04-09T22:02:43.367 DEBUG:boto:Token: None 2024-04-09T22:02:43.367 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.367 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7Y0sHncTxmBZeaiVg85X+LrgV+Y= 2024-04-09T22:02:43.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Y0sHncTxmBZeaiVg85X+LrgV+Y=', 'Content-Length': '0'} 2024-04-09T22:02:43.369 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ec67a8396126e61-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.370 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.370 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.370 DEBUG:boto:path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.370 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.370 DEBUG:boto:Method: GET 2024-04-09T22:02:43.370 DEBUG:boto:Path: /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.370 DEBUG:boto:Data: 2024-04-09T22:02:43.370 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.370 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.371 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.371 DEBUG:boto:Params: {} 2024-04-09T22:02:43.371 DEBUG:boto:Token: None 2024-04-09T22:02:43.371 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.371 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA= 2024-04-09T22:02:43.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA=', 'Content-Length': '0'} 2024-04-09T22:02:43.373 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dc6a58e42252d7a-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.373 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.373 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.373 DEBUG:boto:path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.373 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.373 DEBUG:boto:Method: GET 2024-04-09T22:02:43.374 DEBUG:boto:Path: /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.374 DEBUG:boto:Data: 2024-04-09T22:02:43.374 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.374 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.374 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.374 DEBUG:boto:Params: {} 2024-04-09T22:02:43.374 DEBUG:boto:Token: None 2024-04-09T22:02:43.374 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=bIbsr3kXomqanFnhcQh4V9hg52sTgxn 2024-04-09T22:02:43.374 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA= 2024-04-09T22:02:43.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ehz1wIhmjKtQ7T+v9Tj/YiXWPFA=', 'Content-Length': '0'} 2024-04-09T22:02:43.376 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'bIbsr3kXomqanFnhcQh4V9hg52sTgxn'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005197c96f41f5a305-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.376 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.377 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.377 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.377 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.377 DEBUG:boto:Method: GET 2024-04-09T22:02:43.377 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.377 DEBUG:boto:Data: 2024-04-09T22:02:43.377 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.377 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.377 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.377 DEBUG:boto:Params: {} 2024-04-09T22:02:43.377 DEBUG:boto:Token: None 2024-04-09T22:02:43.378 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.378 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dffb94f5fbf66758-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.380 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.381 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.381 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.381 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.381 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.381 DEBUG:boto:Method: GET 2024-04-09T22:02:43.381 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.381 DEBUG:boto:Data: 2024-04-09T22:02:43.381 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.381 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.382 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.382 DEBUG:boto:Params: {} 2024-04-09T22:02:43.382 DEBUG:boto:Token: None 2024-04-09T22:02:43.382 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.382 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069abbdd96dceef3-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.385 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.385 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.385 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.385 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.385 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.385 DEBUG:boto:Data: 2024-04-09T22:02:43.385 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.385 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.386 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.386 DEBUG:boto:Params: {} 2024-04-09T22:02:43.386 DEBUG:boto:Token: None 2024-04-09T22:02:43.386 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.386 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.389 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0204e087147a9ce-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.390 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.390 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.390 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.390 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.390 DEBUG:boto:Data: 2024-04-09T22:02:43.390 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.390 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.390 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.390 DEBUG:boto:Params: {} 2024-04-09T22:02:43.391 DEBUG:boto:Token: None 2024-04-09T22:02:43.391 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.391 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.394 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4e5961646692b2-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.394 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.394 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.394 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.394 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.394 DEBUG:boto:Method: GET 2024-04-09T22:02:43.395 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.395 DEBUG:boto:Data: 2024-04-09T22:02:43.395 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.395 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.395 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.395 DEBUG:boto:Params: {} 2024-04-09T22:02:43.395 DEBUG:boto:Token: None 2024-04-09T22:02:43.395 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.395 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.397 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003788866cea415ec4-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.438 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.439 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.439 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.439 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.439 DEBUG:boto:Method: GET 2024-04-09T22:02:43.439 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.439 DEBUG:boto:Data: 2024-04-09T22:02:43.439 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.439 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.439 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.439 DEBUG:boto:Params: {} 2024-04-09T22:02:43.440 DEBUG:boto:Token: None 2024-04-09T22:02:43.440 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.440 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.442 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dcedde77959fba4-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.442 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.442 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.442 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.442 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.443 DEBUG:boto:Method: GET 2024-04-09T22:02:43.443 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.443 DEBUG:boto:Data: 2024-04-09T22:02:43.443 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.443 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.443 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.443 DEBUG:boto:Params: {} 2024-04-09T22:02:43.443 DEBUG:boto:Token: None 2024-04-09T22:02:43.443 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.443 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096d05f78b918ea37-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.446 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.447 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.447 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.447 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.447 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.447 DEBUG:boto:Method: GET 2024-04-09T22:02:43.447 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.447 DEBUG:boto:Data: 2024-04-09T22:02:43.447 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.447 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.447 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.447 DEBUG:boto:Params: {} 2024-04-09T22:02:43.448 DEBUG:boto:Token: None 2024-04-09T22:02:43.448 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.448 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb133dd98c398e29-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.451 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.451 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.451 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.451 DEBUG:boto:path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.451 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.451 DEBUG:boto:Method: GET 2024-04-09T22:02:43.451 DEBUG:boto:Path: /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.452 DEBUG:boto:Data: 2024-04-09T22:02:43.452 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.452 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.452 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.452 DEBUG:boto:Params: {} 2024-04-09T22:02:43.452 DEBUG:boto:Token: None 2024-04-09T22:02:43.452 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.452 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY= 2024-04-09T22:02:43.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY=', 'Content-Length': '0'} 2024-04-09T22:02:43.454 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f88d97f7cdb5a6c6-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.454 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.454 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.455 DEBUG:boto:path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.455 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.455 DEBUG:boto:Method: GET 2024-04-09T22:02:43.455 DEBUG:boto:Path: /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.455 DEBUG:boto:Data: 2024-04-09T22:02:43.455 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.455 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.455 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.455 DEBUG:boto:Params: {} 2024-04-09T22:02:43.455 DEBUG:boto:Token: None 2024-04-09T22:02:43.455 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.456 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY= 2024-04-09T22:02:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY=', 'Content-Length': '0'} 2024-04-09T22:02:43.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c053160c9c501ae0-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.458 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.458 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.458 DEBUG:boto:path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.459 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.459 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.459 DEBUG:boto:Path: /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.459 DEBUG:boto:Data: 2024-04-09T22:02:43.459 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.459 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.459 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.459 DEBUG:boto:Params: {} 2024-04-09T22:02:43.459 DEBUG:boto:Token: None 2024-04-09T22:02:43.459 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7uz+bT5sx2JvdLZLnW5zK7YhHIQ= 2024-04-09T22:02:43.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7uz+bT5sx2JvdLZLnW5zK7YhHIQ=', 'Content-Length': '0'} 2024-04-09T22:02:43.461 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cdebfe77c9725a66-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.461 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.462 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.462 DEBUG:boto:path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.462 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.462 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.462 DEBUG:boto:Path: /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.462 DEBUG:boto:Data: 2024-04-09T22:02:43.462 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.462 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.462 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.462 DEBUG:boto:Params: {} 2024-04-09T22:02:43.463 DEBUG:boto:Token: None 2024-04-09T22:02:43.463 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.463 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7uz+bT5sx2JvdLZLnW5zK7YhHIQ= 2024-04-09T22:02:43.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7uz+bT5sx2JvdLZLnW5zK7YhHIQ=', 'Content-Length': '0'} 2024-04-09T22:02:43.465 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4a49b6cd3212f86-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.465 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.465 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.465 DEBUG:boto:path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.465 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.465 DEBUG:boto:Method: GET 2024-04-09T22:02:43.466 DEBUG:boto:Path: /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.466 DEBUG:boto:Data: 2024-04-09T22:02:43.466 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.466 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.466 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.466 DEBUG:boto:Params: {} 2024-04-09T22:02:43.466 DEBUG:boto:Token: None 2024-04-09T22:02:43.466 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.466 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY= 2024-04-09T22:02:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY=', 'Content-Length': '0'} 2024-04-09T22:02:43.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6d67f1ffd11de2a-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.510 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.511 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.511 DEBUG:boto:path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.511 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.511 DEBUG:boto:Method: GET 2024-04-09T22:02:43.511 DEBUG:boto:Path: /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.511 DEBUG:boto:Data: 2024-04-09T22:02:43.512 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.512 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.512 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.512 DEBUG:boto:Params: {} 2024-04-09T22:02:43.513 DEBUG:boto:Token: None 2024-04-09T22:02:43.513 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=3szGQVHc-46oVFHsOBsQWeWjtcNxuwL 2024-04-09T22:02:43.513 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY= 2024-04-09T22:02:43.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:C33LteXBpfFV/9HjYhf/vsi22wY=', 'Content-Length': '0'} 2024-04-09T22:02:43.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '3szGQVHc-46oVFHsOBsQWeWjtcNxuwL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cee9d2cff67ac4da-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.516 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.516 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.516 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.517 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.517 DEBUG:boto:Method: GET 2024-04-09T22:02:43.517 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.517 DEBUG:boto:Data: 2024-04-09T22:02:43.517 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.518 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.518 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.518 DEBUG:boto:Params: {} 2024-04-09T22:02:43.518 DEBUG:boto:Token: None 2024-04-09T22:02:43.518 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.519 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fcfc33ac2f664ce-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.522 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.523 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.523 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.523 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.523 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.523 DEBUG:boto:Method: GET 2024-04-09T22:02:43.524 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.524 DEBUG:boto:Data: 2024-04-09T22:02:43.524 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.524 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.524 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.525 DEBUG:boto:Params: {} 2024-04-09T22:02:43.525 DEBUG:boto:Token: None 2024-04-09T22:02:43.525 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.525 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035404876c9aa9151-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.574 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.575 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.575 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.575 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.575 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.575 DEBUG:boto:Data: 2024-04-09T22:02:43.575 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.575 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.576 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.576 DEBUG:boto:Params: {} 2024-04-09T22:02:43.576 DEBUG:boto:Token: None 2024-04-09T22:02:43.576 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.576 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.579 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079a6be8c7db6bf23-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.580 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.580 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.580 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.580 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.580 DEBUG:boto:Data: 2024-04-09T22:02:43.580 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.580 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.580 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.581 DEBUG:boto:Params: {} 2024-04-09T22:02:43.581 DEBUG:boto:Token: None 2024-04-09T22:02:43.581 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.581 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.584 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000432b954a37ff50e2-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.585 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.585 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.585 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.585 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.585 DEBUG:boto:Method: GET 2024-04-09T22:02:43.585 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.585 DEBUG:boto:Data: 2024-04-09T22:02:43.585 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.585 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.585 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.585 DEBUG:boto:Params: {} 2024-04-09T22:02:43.586 DEBUG:boto:Token: None 2024-04-09T22:02:43.586 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.586 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.588 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090cf724d837712fb-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.634 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.635 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.635 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.635 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.635 DEBUG:boto:Method: GET 2024-04-09T22:02:43.635 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.635 DEBUG:boto:Data: 2024-04-09T22:02:43.635 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.635 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.635 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.635 DEBUG:boto:Params: {} 2024-04-09T22:02:43.636 DEBUG:boto:Token: None 2024-04-09T22:02:43.636 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.636 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.638 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046397343b2982aa3-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.638 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.638 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.638 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.639 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.639 DEBUG:boto:Method: GET 2024-04-09T22:02:43.639 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.639 DEBUG:boto:Data: 2024-04-09T22:02:43.639 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.639 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.639 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.639 DEBUG:boto:Params: {} 2024-04-09T22:02:43.639 DEBUG:boto:Token: None 2024-04-09T22:02:43.639 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.640 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dff42f4db0ce0cc1-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.682 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.683 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.683 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.683 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.683 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.683 DEBUG:boto:Method: GET 2024-04-09T22:02:43.683 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.683 DEBUG:boto:Data: 2024-04-09T22:02:43.683 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.684 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.684 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.684 DEBUG:boto:Params: {} 2024-04-09T22:02:43.684 DEBUG:boto:Token: None 2024-04-09T22:02:43.684 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.684 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a8359ada0dcb59d-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.687 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.687 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.688 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.688 DEBUG:boto:path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.688 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.688 DEBUG:boto:Method: GET 2024-04-09T22:02:43.688 DEBUG:boto:Path: /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.688 DEBUG:boto:Data: 2024-04-09T22:02:43.688 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.688 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.688 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.688 DEBUG:boto:Params: {} 2024-04-09T22:02:43.688 DEBUG:boto:Token: None 2024-04-09T22:02:43.689 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.689 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA= 2024-04-09T22:02:43.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA=', 'Content-Length': '0'} 2024-04-09T22:02:43.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000641ef595c8a02ca7-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.691 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.691 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.691 DEBUG:boto:path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.691 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.691 DEBUG:boto:Method: GET 2024-04-09T22:02:43.692 DEBUG:boto:Path: /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.692 DEBUG:boto:Data: 2024-04-09T22:02:43.692 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.692 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.692 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.692 DEBUG:boto:Params: {} 2024-04-09T22:02:43.692 DEBUG:boto:Token: None 2024-04-09T22:02:43.692 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.692 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA= 2024-04-09T22:02:43.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA=', 'Content-Length': '0'} 2024-04-09T22:02:43.694 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061e8e3dcf9eda18e-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.694 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.695 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.695 DEBUG:boto:path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.695 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.695 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.695 DEBUG:boto:Path: /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.695 DEBUG:boto:Data: 2024-04-09T22:02:43.695 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.695 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.695 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.695 DEBUG:boto:Params: {} 2024-04-09T22:02:43.695 DEBUG:boto:Token: None 2024-04-09T22:02:43.696 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.696 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9KXwfYg93C2W/ld5cSrt4V/AIfQ= 2024-04-09T22:02:43.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9KXwfYg93C2W/ld5cSrt4V/AIfQ=', 'Content-Length': '0'} 2024-04-09T22:02:43.697 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ba2a64cd81f2fe8-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.698 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.698 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.698 DEBUG:boto:path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.698 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.698 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.698 DEBUG:boto:Path: /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.698 DEBUG:boto:Data: 2024-04-09T22:02:43.698 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.698 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.699 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.699 DEBUG:boto:Params: {} 2024-04-09T22:02:43.699 DEBUG:boto:Token: None 2024-04-09T22:02:43.699 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.699 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9KXwfYg93C2W/ld5cSrt4V/AIfQ= 2024-04-09T22:02:43.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9KXwfYg93C2W/ld5cSrt4V/AIfQ=', 'Content-Length': '0'} 2024-04-09T22:02:43.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b0bf66f7091aeb7-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.701 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.701 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.701 DEBUG:boto:path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.701 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.701 DEBUG:boto:Method: GET 2024-04-09T22:02:43.701 DEBUG:boto:Path: /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.702 DEBUG:boto:Data: 2024-04-09T22:02:43.702 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.702 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.702 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.702 DEBUG:boto:Params: {} 2024-04-09T22:02:43.702 DEBUG:boto:Token: None 2024-04-09T22:02:43.702 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.702 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA= 2024-04-09T22:02:43.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA=', 'Content-Length': '0'} 2024-04-09T22:02:43.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007357faf0dcd99b38-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.746 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.747 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.747 DEBUG:boto:path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.747 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.747 DEBUG:boto:Method: GET 2024-04-09T22:02:43.747 DEBUG:boto:Path: /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.747 DEBUG:boto:Data: 2024-04-09T22:02:43.747 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.747 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.747 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.747 DEBUG:boto:Params: {} 2024-04-09T22:02:43.748 DEBUG:boto:Token: None 2024-04-09T22:02:43.748 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=QCLRNCVITG24MAEWwoGf.BsQb23bjCe 2024-04-09T22:02:43.748 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA= 2024-04-09T22:02:43.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t8SiknZ4JEwZChKjNeXKF+n9OuA=', 'Content-Length': '0'} 2024-04-09T22:02:43.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'QCLRNCVITG24MAEWwoGf.BsQb23bjCe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008626a5a80df32fb5-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.750 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.751 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.751 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.751 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.751 DEBUG:boto:Method: GET 2024-04-09T22:02:43.751 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.751 DEBUG:boto:Data: 2024-04-09T22:02:43.751 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.751 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.751 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.751 DEBUG:boto:Params: {} 2024-04-09T22:02:43.751 DEBUG:boto:Token: None 2024-04-09T22:02:43.752 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.752 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.754 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbd5a88ea13b87c-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.802 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.803 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.803 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.803 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.803 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.803 DEBUG:boto:Method: GET 2024-04-09T22:02:43.803 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.803 DEBUG:boto:Data: 2024-04-09T22:02:43.804 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.804 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.804 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.804 DEBUG:boto:Params: {} 2024-04-09T22:02:43.804 DEBUG:boto:Token: None 2024-04-09T22:02:43.804 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020dcd8e602d5097d-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.807 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.807 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.807 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.808 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.808 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.808 DEBUG:boto:Data: 2024-04-09T22:02:43.808 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.808 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.808 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.808 DEBUG:boto:Params: {} 2024-04-09T22:02:43.808 DEBUG:boto:Token: None 2024-04-09T22:02:43.808 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.808 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000565e5e13c16cccf8-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.815 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.815 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.815 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.815 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:43.815 DEBUG:boto:Data: 2024-04-09T22:02:43.815 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.815 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.815 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.815 DEBUG:boto:Params: {} 2024-04-09T22:02:43.816 DEBUG:boto:Token: None 2024-04-09T22:02:43.816 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj 2024-04-09T22:02:43.816 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc= 2024-04-09T22:02:43.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yh407/qKWtY97eVA+VlC31sGNhc=', 'Content-Length': '0'} 2024-04-09T22:02:43.819 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000045af90aafbc8a081-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.820 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.820 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.820 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.820 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.820 DEBUG:boto:Method: GET 2024-04-09T22:02:43.820 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.820 DEBUG:boto:Data: 2024-04-09T22:02:43.820 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.820 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.821 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.821 DEBUG:boto:Params: {} 2024-04-09T22:02:43.821 DEBUG:boto:Token: None 2024-04-09T22:02:43.821 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.821 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea057df5750ecba5-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.866 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.867 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.867 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.867 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.867 DEBUG:boto:Method: GET 2024-04-09T22:02:43.867 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.867 DEBUG:boto:Data: 2024-04-09T22:02:43.867 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.867 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.867 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.867 DEBUG:boto:Params: {} 2024-04-09T22:02:43.868 DEBUG:boto:Token: None 2024-04-09T22:02:43.868 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:43.868 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk= 2024-04-09T22:02:43.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UVBwH3rKzopWO6xP4OgCn+SxXQk=', 'Content-Length': '0'} 2024-04-09T22:02:43.870 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7702609e9e4e63d-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.870 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.871 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.871 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.871 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.871 DEBUG:boto:Method: GET 2024-04-09T22:02:43.871 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.871 DEBUG:boto:Data: 2024-04-09T22:02:43.871 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.871 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.871 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.871 DEBUG:boto:Params: {} 2024-04-09T22:02:43.872 DEBUG:boto:Token: None 2024-04-09T22:02:43.872 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.872 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1dcee96d830e46-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.922 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.923 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.923 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.923 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.923 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.923 DEBUG:boto:Method: GET 2024-04-09T22:02:43.923 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.923 DEBUG:boto:Data: 2024-04-09T22:02:43.924 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.924 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.924 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.924 DEBUG:boto:Params: {} 2024-04-09T22:02:43.924 DEBUG:boto:Token: None 2024-04-09T22:02:43.924 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.924 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006146214d8151f4ec-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.927 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.928 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.928 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.928 DEBUG:boto:path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.928 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.928 DEBUG:boto:Method: GET 2024-04-09T22:02:43.928 DEBUG:boto:Path: /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.928 DEBUG:boto:Data: 2024-04-09T22:02:43.928 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.928 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.928 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.929 DEBUG:boto:Params: {} 2024-04-09T22:02:43.929 DEBUG:boto:Token: None 2024-04-09T22:02:43.929 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.929 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo= 2024-04-09T22:02:43.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo=', 'Content-Length': '0'} 2024-04-09T22:02:43.931 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f55631560645f28-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.931 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.931 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.931 DEBUG:boto:path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.931 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.931 DEBUG:boto:Method: GET 2024-04-09T22:02:43.932 DEBUG:boto:Path: /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.932 DEBUG:boto:Data: 2024-04-09T22:02:43.932 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.932 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.932 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.932 DEBUG:boto:Params: {} 2024-04-09T22:02:43.932 DEBUG:boto:Token: None 2024-04-09T22:02:43.932 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.932 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo= 2024-04-09T22:02:43.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo=', 'Content-Length': '0'} 2024-04-09T22:02:43.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000126054b49b32b81-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.935 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.935 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.935 DEBUG:boto:path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.935 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.935 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.935 DEBUG:boto:Path: /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.936 DEBUG:boto:Data: 2024-04-09T22:02:43.936 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.936 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.936 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.936 DEBUG:boto:Params: {} 2024-04-09T22:02:43.936 DEBUG:boto:Token: None 2024-04-09T22:02:43.936 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.936 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PzyQIff5cpnjKhEvnESWZeQbu9Y= 2024-04-09T22:02:43.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PzyQIff5cpnjKhEvnESWZeQbu9Y=', 'Content-Length': '0'} 2024-04-09T22:02:43.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007e02db806327fe24-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.938 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.938 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.938 DEBUG:boto:path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.939 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.939 DEBUG:boto:Method: HEAD 2024-04-09T22:02:43.939 DEBUG:boto:Path: /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.939 DEBUG:boto:Data: 2024-04-09T22:02:43.939 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.939 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.939 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.939 DEBUG:boto:Params: {} 2024-04-09T22:02:43.939 DEBUG:boto:Token: None 2024-04-09T22:02:43.939 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.940 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PzyQIff5cpnjKhEvnESWZeQbu9Y= 2024-04-09T22:02:43.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PzyQIff5cpnjKhEvnESWZeQbu9Y=', 'Content-Length': '0'} 2024-04-09T22:02:43.941 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b5fa07638c1116c-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:43.942 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.942 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.942 DEBUG:boto:path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.942 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.942 DEBUG:boto:Method: GET 2024-04-09T22:02:43.942 DEBUG:boto:Path: /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.942 DEBUG:boto:Data: 2024-04-09T22:02:43.942 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.942 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.942 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.943 DEBUG:boto:Params: {} 2024-04-09T22:02:43.943 DEBUG:boto:Token: None 2024-04-09T22:02:43.943 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.943 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo= 2024-04-09T22:02:43.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo=', 'Content-Length': '0'} 2024-04-09T22:02:43.945 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e4b4306c50e55bf-006615bb03-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.945 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.945 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:43.945 DEBUG:boto:path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.945 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.946 DEBUG:boto:Method: GET 2024-04-09T22:02:43.946 DEBUG:boto:Path: /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.946 DEBUG:boto:Data: 2024-04-09T22:02:43.946 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.946 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.946 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.946 DEBUG:boto:Params: {} 2024-04-09T22:02:43.946 DEBUG:boto:Token: None 2024-04-09T22:02:43.946 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/obj?versionId=0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx 2024-04-09T22:02:43.946 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo= 2024-04-09T22:02:43.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1tEUhcfyICVTY+ewbRtSWy4VpSo=', 'Content-Length': '0'} 2024-04-09T22:02:43.948 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', '0NqfrL7s1QfDcsYG46xsNqMKsn8x8Qx'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006514143dfe2b43d8-006615bb03-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.990 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.991 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.991 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.991 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.991 DEBUG:boto:Method: GET 2024-04-09T22:02:43.991 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.991 DEBUG:boto:Data: 2024-04-09T22:02:43.991 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.991 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:43.991 DEBUG:boto:Port: 8001 2024-04-09T22:02:43.991 DEBUG:boto:Params: {} 2024-04-09T22:02:43.992 DEBUG:boto:Token: None 2024-04-09T22:02:43.992 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.992 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492a86d1e949c059-006615bb03-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.995 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.995 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:43.995 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:43.995 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.996 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:43.996 DEBUG:boto:Method: GET 2024-04-09T22:02:43.996 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:43.996 DEBUG:boto:Data: 2024-04-09T22:02:43.996 DEBUG:boto:Headers: {} 2024-04-09T22:02:43.996 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:43.996 DEBUG:boto:Port: 8000 2024-04-09T22:02:43.996 DEBUG:boto:Params: {} 2024-04-09T22:02:43.996 DEBUG:boto:Token: None 2024-04-09T22:02:43.996 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:43 GMT /xryqqk-15/ 2024-04-09T22:02:43.997 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI= 2024-04-09T22:02:43.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lm7Zn86fsxSX0elM0OPz4zDxIUI=', 'Content-Length': '0'} 2024-04-09T22:02:43.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb425fe918484b8-006615bb03-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:43.999 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:43.999 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:43.999 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.000 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.000 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.000 DEBUG:boto:Data: 2024-04-09T22:02:44.000 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.000 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.000 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.000 DEBUG:boto:Params: {} 2024-04-09T22:02:44.000 DEBUG:boto:Token: None 2024-04-09T22:02:44.000 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.000 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.003 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000090df4709a1e6c563-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.004 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.004 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.004 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.004 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.004 DEBUG:boto:Data: 2024-04-09T22:02:44.004 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.004 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.004 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.004 DEBUG:boto:Params: {} 2024-04-09T22:02:44.005 DEBUG:boto:Token: None 2024-04-09T22:02:44.005 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.005 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ed9e3165a304fdb5-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.008 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.008 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.009 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.009 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.009 DEBUG:boto:Method: GET 2024-04-09T22:02:44.009 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.009 DEBUG:boto:Data: 2024-04-09T22:02:44.009 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.009 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.009 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.009 DEBUG:boto:Params: {} 2024-04-09T22:02:44.009 DEBUG:boto:Token: None 2024-04-09T22:02:44.009 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.010 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.011 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a89b83d4517278f-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.058 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.059 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.059 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.059 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.059 DEBUG:boto:Method: GET 2024-04-09T22:02:44.059 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.059 DEBUG:boto:Data: 2024-04-09T22:02:44.059 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.059 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.059 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.059 DEBUG:boto:Params: {} 2024-04-09T22:02:44.060 DEBUG:boto:Token: None 2024-04-09T22:02:44.060 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.060 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.062 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce6b66df20b33dae-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.062 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.063 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.063 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.063 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.063 DEBUG:boto:Method: GET 2024-04-09T22:02:44.063 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.063 DEBUG:boto:Data: 2024-04-09T22:02:44.063 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.063 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.063 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.063 DEBUG:boto:Params: {} 2024-04-09T22:02:44.063 DEBUG:boto:Token: None 2024-04-09T22:02:44.064 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.064 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb5c68daf2623445-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.066 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.067 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.067 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.067 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.067 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.067 DEBUG:boto:Method: GET 2024-04-09T22:02:44.067 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.067 DEBUG:boto:Data: 2024-04-09T22:02:44.067 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.067 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.068 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.068 DEBUG:boto:Params: {} 2024-04-09T22:02:44.068 DEBUG:boto:Token: None 2024-04-09T22:02:44.069 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.069 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.071 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee95ef4eb0d01f0d-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.072 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.072 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.072 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.072 DEBUG:boto:path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.073 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.073 DEBUG:boto:Method: GET 2024-04-09T22:02:44.073 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.073 DEBUG:boto:Data: 2024-04-09T22:02:44.073 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.073 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.073 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.073 DEBUG:boto:Params: {} 2024-04-09T22:02:44.073 DEBUG:boto:Token: None 2024-04-09T22:02:44.073 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.074 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms= 2024-04-09T22:02:44.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms=', 'Content-Length': '0'} 2024-04-09T22:02:44.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006ea55f319b85928b-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.076 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.076 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.076 DEBUG:boto:path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.076 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.076 DEBUG:boto:Method: GET 2024-04-09T22:02:44.076 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.077 DEBUG:boto:Data: 2024-04-09T22:02:44.077 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.077 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.077 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.077 DEBUG:boto:Params: {} 2024-04-09T22:02:44.077 DEBUG:boto:Token: None 2024-04-09T22:02:44.077 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.077 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms= 2024-04-09T22:02:44.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms=', 'Content-Length': '0'} 2024-04-09T22:02:44.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000884c61687beef90e-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.126 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.126 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.126 DEBUG:boto:path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.127 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.127 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.127 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.127 DEBUG:boto:Data: 2024-04-09T22:02:44.127 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.127 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.127 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.127 DEBUG:boto:Params: {} 2024-04-09T22:02:44.127 DEBUG:boto:Token: None 2024-04-09T22:02:44.127 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.128 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/qktMEA5wObElMiUoZh6zswZt9Y= 2024-04-09T22:02:44.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/qktMEA5wObElMiUoZh6zswZt9Y=', 'Content-Length': '0'} 2024-04-09T22:02:44.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a4a98539d82fa535-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.130 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.130 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.130 DEBUG:boto:path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.130 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.130 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.130 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.130 DEBUG:boto:Data: 2024-04-09T22:02:44.130 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.131 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.131 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.131 DEBUG:boto:Params: {} 2024-04-09T22:02:44.131 DEBUG:boto:Token: None 2024-04-09T22:02:44.131 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.131 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/qktMEA5wObElMiUoZh6zswZt9Y= 2024-04-09T22:02:44.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/qktMEA5wObElMiUoZh6zswZt9Y=', 'Content-Length': '0'} 2024-04-09T22:02:44.133 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7c9a3924f00098e-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.133 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.133 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.133 DEBUG:boto:path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.133 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.133 DEBUG:boto:Method: GET 2024-04-09T22:02:44.134 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.134 DEBUG:boto:Data: 2024-04-09T22:02:44.134 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.134 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.134 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.134 DEBUG:boto:Params: {} 2024-04-09T22:02:44.134 DEBUG:boto:Token: None 2024-04-09T22:02:44.134 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.134 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms= 2024-04-09T22:02:44.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms=', 'Content-Length': '0'} 2024-04-09T22:02:44.137 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f6dcb1a44adb43c-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.178 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.179 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.179 DEBUG:boto:path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.179 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.179 DEBUG:boto:Method: GET 2024-04-09T22:02:44.179 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.179 DEBUG:boto:Data: 2024-04-09T22:02:44.179 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.180 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.180 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.180 DEBUG:boto:Params: {} 2024-04-09T22:02:44.180 DEBUG:boto:Token: None 2024-04-09T22:02:44.180 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo 2024-04-09T22:02:44.180 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms= 2024-04-09T22:02:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5pfnVaWt4mKsBZBJ52ZDb6fntms=', 'Content-Length': '0'} 2024-04-09T22:02:44.182 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Z.sV.ddadxTWP-UmBEgRWXYXeri-9vo'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020b29152d3fbde2e-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.183 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.183 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.183 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.183 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.183 DEBUG:boto:Method: GET 2024-04-09T22:02:44.183 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.183 DEBUG:boto:Data: 2024-04-09T22:02:44.183 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.183 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.184 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.184 DEBUG:boto:Params: {} 2024-04-09T22:02:44.185 DEBUG:boto:Token: None 2024-04-09T22:02:44.185 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.185 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bafb9906cc30081-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.231 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.231 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.231 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.232 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.232 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.232 DEBUG:boto:Method: GET 2024-04-09T22:02:44.232 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.232 DEBUG:boto:Data: 2024-04-09T22:02:44.232 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.232 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.232 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.232 DEBUG:boto:Params: {} 2024-04-09T22:02:44.232 DEBUG:boto:Token: None 2024-04-09T22:02:44.233 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.233 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.235 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c20b556a63ff462a-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.236 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.236 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.236 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.236 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.236 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.236 DEBUG:boto:Data: 2024-04-09T22:02:44.236 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.237 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.237 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.237 DEBUG:boto:Params: {} 2024-04-09T22:02:44.237 DEBUG:boto:Token: None 2024-04-09T22:02:44.237 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.237 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.240 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005a4f611eb9a65331-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.241 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.241 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.241 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.241 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.241 DEBUG:boto:Data: 2024-04-09T22:02:44.241 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.241 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.241 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.241 DEBUG:boto:Params: {} 2024-04-09T22:02:44.241 DEBUG:boto:Token: None 2024-04-09T22:02:44.242 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.242 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.245 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce765c7bbf8d5276-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.246 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.246 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.246 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.246 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.246 DEBUG:boto:Method: GET 2024-04-09T22:02:44.246 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.246 DEBUG:boto:Data: 2024-04-09T22:02:44.246 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.246 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.246 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.246 DEBUG:boto:Params: {} 2024-04-09T22:02:44.247 DEBUG:boto:Token: None 2024-04-09T22:02:44.247 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.247 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.249 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a09ee767be5ddb3-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.290 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.291 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.291 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.291 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.291 DEBUG:boto:Method: GET 2024-04-09T22:02:44.291 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.291 DEBUG:boto:Data: 2024-04-09T22:02:44.291 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.291 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.291 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.291 DEBUG:boto:Params: {} 2024-04-09T22:02:44.292 DEBUG:boto:Token: None 2024-04-09T22:02:44.292 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.292 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.294 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095a2e4ef9c3f20c2-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.294 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.294 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.295 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.295 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.295 DEBUG:boto:Method: GET 2024-04-09T22:02:44.295 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.295 DEBUG:boto:Data: 2024-04-09T22:02:44.295 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.295 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.295 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.295 DEBUG:boto:Params: {} 2024-04-09T22:02:44.295 DEBUG:boto:Token: None 2024-04-09T22:02:44.296 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.296 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2fe7c157711a381-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.338 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.339 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.339 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.339 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.339 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.339 DEBUG:boto:Method: GET 2024-04-09T22:02:44.339 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.339 DEBUG:boto:Data: 2024-04-09T22:02:44.339 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.340 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.340 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.340 DEBUG:boto:Params: {} 2024-04-09T22:02:44.340 DEBUG:boto:Token: None 2024-04-09T22:02:44.340 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.340 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008bbc218e298fb7eb-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.343 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.343 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.344 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.344 DEBUG:boto:path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.344 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.344 DEBUG:boto:Method: GET 2024-04-09T22:02:44.344 DEBUG:boto:Path: /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.344 DEBUG:boto:Data: 2024-04-09T22:02:44.344 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.344 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.344 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.344 DEBUG:boto:Params: {} 2024-04-09T22:02:44.345 DEBUG:boto:Token: None 2024-04-09T22:02:44.345 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.345 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A= 2024-04-09T22:02:44.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A=', 'Content-Length': '0'} 2024-04-09T22:02:44.347 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bb69177af36787fa-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.347 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.347 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.347 DEBUG:boto:path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.347 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.347 DEBUG:boto:Method: GET 2024-04-09T22:02:44.348 DEBUG:boto:Path: /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.348 DEBUG:boto:Data: 2024-04-09T22:02:44.348 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.348 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.348 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.348 DEBUG:boto:Params: {} 2024-04-09T22:02:44.348 DEBUG:boto:Token: None 2024-04-09T22:02:44.348 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.348 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A= 2024-04-09T22:02:44.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A=', 'Content-Length': '0'} 2024-04-09T22:02:44.350 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c10e47159c83e33a-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.399 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.399 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.399 DEBUG:boto:path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.399 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.399 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.399 DEBUG:boto:Path: /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.399 DEBUG:boto:Data: 2024-04-09T22:02:44.399 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.399 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.399 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.400 DEBUG:boto:Params: {} 2024-04-09T22:02:44.400 DEBUG:boto:Token: None 2024-04-09T22:02:44.400 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.400 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+v0lzFK/Wd6euh83Vnjpvx6Zs4= 2024-04-09T22:02:44.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+v0lzFK/Wd6euh83Vnjpvx6Zs4=', 'Content-Length': '0'} 2024-04-09T22:02:44.402 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a15ee2b6b3739ef7-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.402 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.402 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.403 DEBUG:boto:path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.403 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.403 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.403 DEBUG:boto:Path: /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.403 DEBUG:boto:Data: 2024-04-09T22:02:44.403 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.403 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.403 DEBUG:boto:Params: {} 2024-04-09T22:02:44.403 DEBUG:boto:Token: None 2024-04-09T22:02:44.404 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+v0lzFK/Wd6euh83Vnjpvx6Zs4= 2024-04-09T22:02:44.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+v0lzFK/Wd6euh83Vnjpvx6Zs4=', 'Content-Length': '0'} 2024-04-09T22:02:44.405 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f504683214c1b78-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.406 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.406 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.406 DEBUG:boto:path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.406 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.406 DEBUG:boto:Method: GET 2024-04-09T22:02:44.407 DEBUG:boto:Path: /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.407 DEBUG:boto:Data: 2024-04-09T22:02:44.407 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.407 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.407 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.407 DEBUG:boto:Params: {} 2024-04-09T22:02:44.407 DEBUG:boto:Token: None 2024-04-09T22:02:44.407 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.407 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A= 2024-04-09T22:02:44.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A=', 'Content-Length': '0'} 2024-04-09T22:02:44.410 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab6d94ef20e6171d-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.454 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.455 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.455 DEBUG:boto:path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.455 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.455 DEBUG:boto:Method: GET 2024-04-09T22:02:44.455 DEBUG:boto:Path: /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.455 DEBUG:boto:Data: 2024-04-09T22:02:44.455 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.455 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.455 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.455 DEBUG:boto:Params: {} 2024-04-09T22:02:44.456 DEBUG:boto:Token: None 2024-04-09T22:02:44.456 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc 2024-04-09T22:02:44.456 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A= 2024-04-09T22:02:44.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SAEBAM9Tn2d85pi/pm5F57uW+7A=', 'Content-Length': '0'} 2024-04-09T22:02:44.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'uHIeCAUenOwzrzlgUIRrkkwqPwkwVvc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001162559458380de8-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.458 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.458 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.459 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.459 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.459 DEBUG:boto:Method: GET 2024-04-09T22:02:44.459 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.459 DEBUG:boto:Data: 2024-04-09T22:02:44.460 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.460 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.460 DEBUG:boto:Params: {} 2024-04-09T22:02:44.460 DEBUG:boto:Token: None 2024-04-09T22:02:44.460 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b8cd817894ce863-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.463 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.463 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.464 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.464 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.464 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.464 DEBUG:boto:Method: GET 2024-04-09T22:02:44.464 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.464 DEBUG:boto:Data: 2024-04-09T22:02:44.464 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.464 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.464 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.464 DEBUG:boto:Params: {} 2024-04-09T22:02:44.464 DEBUG:boto:Token: None 2024-04-09T22:02:44.465 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.465 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008308ff937b65306-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.468 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.468 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.468 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.468 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.468 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.468 DEBUG:boto:Data: 2024-04-09T22:02:44.468 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.469 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.469 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.469 DEBUG:boto:Params: {} 2024-04-09T22:02:44.469 DEBUG:boto:Token: None 2024-04-09T22:02:44.469 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.470 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000661c7a0616adc71-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.473 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.474 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.474 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.474 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.474 DEBUG:boto:Data: 2024-04-09T22:02:44.474 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.474 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.474 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.474 DEBUG:boto:Params: {} 2024-04-09T22:02:44.475 DEBUG:boto:Token: None 2024-04-09T22:02:44.475 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.475 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.478 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c27f5e4fe6174978-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.478 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.478 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.479 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.479 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.479 DEBUG:boto:Method: GET 2024-04-09T22:02:44.479 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.479 DEBUG:boto:Data: 2024-04-09T22:02:44.479 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.479 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.479 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.479 DEBUG:boto:Params: {} 2024-04-09T22:02:44.479 DEBUG:boto:Token: None 2024-04-09T22:02:44.480 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.480 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.481 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008bc061f5d7bf8e00-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.522 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.523 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.523 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.523 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.523 DEBUG:boto:Method: GET 2024-04-09T22:02:44.523 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.523 DEBUG:boto:Data: 2024-04-09T22:02:44.523 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.523 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.523 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.523 DEBUG:boto:Params: {} 2024-04-09T22:02:44.524 DEBUG:boto:Token: None 2024-04-09T22:02:44.524 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.524 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.526 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b2337799d933dd1-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.526 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.526 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.526 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.526 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.526 DEBUG:boto:Method: GET 2024-04-09T22:02:44.526 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.527 DEBUG:boto:Data: 2024-04-09T22:02:44.527 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.527 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.527 DEBUG:boto:Params: {} 2024-04-09T22:02:44.527 DEBUG:boto:Token: None 2024-04-09T22:02:44.527 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.527 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f135936223f7140-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.530 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.531 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.531 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.531 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.531 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.531 DEBUG:boto:Method: GET 2024-04-09T22:02:44.531 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.531 DEBUG:boto:Data: 2024-04-09T22:02:44.531 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.532 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.532 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.532 DEBUG:boto:Params: {} 2024-04-09T22:02:44.532 DEBUG:boto:Token: None 2024-04-09T22:02:44.532 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000143e7929520f98e9-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.535 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.536 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.536 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.536 DEBUG:boto:path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.536 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.536 DEBUG:boto:Method: GET 2024-04-09T22:02:44.536 DEBUG:boto:Path: /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.536 DEBUG:boto:Data: 2024-04-09T22:02:44.536 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.536 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.537 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.537 DEBUG:boto:Params: {} 2024-04-09T22:02:44.537 DEBUG:boto:Token: None 2024-04-09T22:02:44.537 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.537 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww= 2024-04-09T22:02:44.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww=', 'Content-Length': '0'} 2024-04-09T22:02:44.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000033defa9098ad17c5-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.539 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.539 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.539 DEBUG:boto:path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.539 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.539 DEBUG:boto:Method: GET 2024-04-09T22:02:44.540 DEBUG:boto:Path: /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.540 DEBUG:boto:Data: 2024-04-09T22:02:44.540 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.540 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.540 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.540 DEBUG:boto:Params: {} 2024-04-09T22:02:44.540 DEBUG:boto:Token: None 2024-04-09T22:02:44.540 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww= 2024-04-09T22:02:44.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww=', 'Content-Length': '0'} 2024-04-09T22:02:44.542 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d994174c2cbe5148-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.587 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.587 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.587 DEBUG:boto:path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.587 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.587 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.587 DEBUG:boto:Path: /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.587 DEBUG:boto:Data: 2024-04-09T22:02:44.587 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.588 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.588 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.588 DEBUG:boto:Params: {} 2024-04-09T22:02:44.588 DEBUG:boto:Token: None 2024-04-09T22:02:44.588 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.588 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PwSzlxHKes07rY7nyjd1voM6P4c= 2024-04-09T22:02:44.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PwSzlxHKes07rY7nyjd1voM6P4c=', 'Content-Length': '0'} 2024-04-09T22:02:44.590 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b27a52a02fa68682-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.590 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.591 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.591 DEBUG:boto:path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.591 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.591 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.591 DEBUG:boto:Path: /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.591 DEBUG:boto:Data: 2024-04-09T22:02:44.591 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.591 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.591 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.591 DEBUG:boto:Params: {} 2024-04-09T22:02:44.591 DEBUG:boto:Token: None 2024-04-09T22:02:44.592 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PwSzlxHKes07rY7nyjd1voM6P4c= 2024-04-09T22:02:44.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PwSzlxHKes07rY7nyjd1voM6P4c=', 'Content-Length': '0'} 2024-04-09T22:02:44.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5b0912a5f5683ef-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.594 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.594 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.594 DEBUG:boto:path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.594 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.594 DEBUG:boto:Method: GET 2024-04-09T22:02:44.594 DEBUG:boto:Path: /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.594 DEBUG:boto:Data: 2024-04-09T22:02:44.594 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.595 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.595 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.595 DEBUG:boto:Params: {} 2024-04-09T22:02:44.595 DEBUG:boto:Token: None 2024-04-09T22:02:44.595 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.595 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww= 2024-04-09T22:02:44.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww=', 'Content-Length': '0'} 2024-04-09T22:02:44.597 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b02f5c4b2a0f26e-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.597 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.597 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.598 DEBUG:boto:path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.598 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.598 DEBUG:boto:Method: GET 2024-04-09T22:02:44.598 DEBUG:boto:Path: /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.598 DEBUG:boto:Data: 2024-04-09T22:02:44.598 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.598 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.598 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.598 DEBUG:boto:Params: {} 2024-04-09T22:02:44.598 DEBUG:boto:Token: None 2024-04-09T22:02:44.599 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL 2024-04-09T22:02:44.599 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww= 2024-04-09T22:02:44.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y2/feGAfcDQJC0Uk4/k2LRLWYww=', 'Content-Length': '0'} 2024-04-09T22:02:44.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'zQzhh-6iBfnwpGd2KJmyy3OGGGRyItL'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e98b07f7a747b5bb-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.642 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.643 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.643 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.643 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.643 DEBUG:boto:Method: GET 2024-04-09T22:02:44.643 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.643 DEBUG:boto:Data: 2024-04-09T22:02:44.643 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.643 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.643 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.643 DEBUG:boto:Params: {} 2024-04-09T22:02:44.644 DEBUG:boto:Token: None 2024-04-09T22:02:44.644 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.644 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a43b4accd06cfa42-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.647 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.647 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.648 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.648 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.648 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.648 DEBUG:boto:Method: GET 2024-04-09T22:02:44.648 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.648 DEBUG:boto:Data: 2024-04-09T22:02:44.648 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.648 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.648 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.648 DEBUG:boto:Params: {} 2024-04-09T22:02:44.648 DEBUG:boto:Token: None 2024-04-09T22:02:44.649 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.649 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000561465d0d200c04e-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.651 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.652 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.652 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.652 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.652 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.652 DEBUG:boto:Data: 2024-04-09T22:02:44.652 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.652 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.652 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.653 DEBUG:boto:Params: {} 2024-04-09T22:02:44.653 DEBUG:boto:Token: None 2024-04-09T22:02:44.653 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.653 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000874a8b8ec0f89901-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.657 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.657 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.657 DEBUG:boto:Method: HEAD 2024-04-09T22:02:44.658 DEBUG:boto:Path: /xryqqk-15/obj 2024-04-09T22:02:44.658 DEBUG:boto:Data: 2024-04-09T22:02:44.658 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.658 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.658 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.658 DEBUG:boto:Params: {} 2024-04-09T22:02:44.658 DEBUG:boto:Token: None 2024-04-09T22:02:44.658 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj 2024-04-09T22:02:44.658 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo= 2024-04-09T22:02:44.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DbpDLFrtjhDe7JdzVQZj1gEr7lo=', 'Content-Length': '0'} 2024-04-09T22:02:44.662 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086c19c3a06eb2c88-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:02:44.662 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.662 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.663 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.663 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.663 DEBUG:boto:Method: GET 2024-04-09T22:02:44.663 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.663 DEBUG:boto:Data: 2024-04-09T22:02:44.663 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.663 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.663 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.663 DEBUG:boto:Params: {} 2024-04-09T22:02:44.663 DEBUG:boto:Token: None 2024-04-09T22:02:44.664 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.664 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-15:a5723776-f142-428c-a6df-d7bf949257da.4372.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d4c4b0b8275fa1e-006615bb04-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.706 DEBUG:boto:path=/xryqqk-15/obj 2024-04-09T22:02:44.707 DEBUG:boto:auth_path=/xryqqk-15/obj 2024-04-09T22:02:44.707 DEBUG:boto:path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.707 DEBUG:boto:auth_path=/xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.707 DEBUG:boto:Method: GET 2024-04-09T22:02:44.707 DEBUG:boto:Path: /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.707 DEBUG:boto:Data: 2024-04-09T22:02:44.707 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.707 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.707 DEBUG:boto:Params: {} 2024-04-09T22:02:44.708 DEBUG:boto:Token: None 2024-04-09T22:02:44.708 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/obj?versionId=Krgu-QkcVxiHciJpP8vdr7GSowJPRVU 2024-04-09T22:02:44.708 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY= 2024-04-09T22:02:44.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yB8MAQtwrSJhKg3wXA17BFHSxTY=', 'Content-Length': '0'} 2024-04-09T22:02:44.710 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:00:00 GMT'), ('x-amz-version-id', 'Krgu-QkcVxiHciJpP8vdr7GSowJPRVU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3627d01c073f45d-006615bb04-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.710 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.710 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.710 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.710 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.711 DEBUG:boto:Method: GET 2024-04-09T22:02:44.711 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.711 DEBUG:boto:Data: 2024-04-09T22:02:44.711 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.711 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:02:44.711 DEBUG:boto:Port: 8001 2024-04-09T22:02:44.711 DEBUG:boto:Params: {} 2024-04-09T22:02:44.711 DEBUG:boto:Token: None 2024-04-09T22:02:44.711 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.711 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c10ff24cf7b14ba-006615bb04-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.714 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.715 DEBUG:boto:path=/xryqqk-15/ 2024-04-09T22:02:44.715 DEBUG:boto:auth_path=/xryqqk-15/ 2024-04-09T22:02:44.715 DEBUG:boto:path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.715 DEBUG:boto:auth_path=/xryqqk-15/?prefix=obj 2024-04-09T22:02:44.715 DEBUG:boto:Method: GET 2024-04-09T22:02:44.715 DEBUG:boto:Path: /xryqqk-15/?prefix=obj 2024-04-09T22:02:44.715 DEBUG:boto:Data: 2024-04-09T22:02:44.715 DEBUG:boto:Headers: {} 2024-04-09T22:02:44.715 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:44.716 DEBUG:boto:Port: 8000 2024-04-09T22:02:44.716 DEBUG:boto:Params: {} 2024-04-09T22:02:44.716 DEBUG:boto:Token: None 2024-04-09T22:02:44.716 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:02:44 GMT /xryqqk-15/ 2024-04-09T22:02:44.716 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M= 2024-04-09T22:02:44.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CRcyg/woBMVWACgc9ubI9evuu4M=', 'Content-Length': '0'} 2024-04-09T22:02:44.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005889b6871037396f-006615bb04-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:02:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:44.719 DEBUG:boto:b'xryqqk-15obj1000falseobj2024-04-09T22:00:00.522Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:02:44.719 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-15 zones={a2, a1} 2024-04-09T22:02:44.719 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-09T22:02:44.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:44.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:44.720 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:44.776 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:44.776 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:44.829 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:02:44.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:48:49.746822Z" 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000003888", 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:00:55.767627Z" 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:00:11.764065Z" 2024-04-09T22:02:44.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.831 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-09T22:02:44.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:48:27.744609Z" 2024-04-09T22:02:44.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:44.831 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:02:44.832 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001918', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000004460', 3: '00000000000000000000:00000000000000002622'} 2024-04-09T22:02:44.832 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T22:02:44.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:44.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:44.832 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:44.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:44.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 15574053103777562020 2024-04-09T22:02:44.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:48:49.746822Z" 2024-04-09T22:02:44.929 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000003888", 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:44.930 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:00:55.767627Z" 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000004460", 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:00:11.764065Z" 2024-04-09T22:02:44.931 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:48:27.744609Z" 2024-04-09T22:02:44.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 0, 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 0 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:02:44Z" 2024-04-09T22:02:44.933 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:02:44.935 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15574053103777562020 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:48:49.746822Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003888", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:00:55.767627Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004460", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:00:11.764065Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:48:27.744609Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-09T22:02:44Z" } 2024-04-09T22:02:44.935 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:48:49.746822Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003888', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:00:55.767627Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004460', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:00:11.764065Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:48:27.744609Z'}}] 2024-04-09T22:02:44.936 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001918', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000004460', 3: '00000000000000000000:00000000000000002622'} 2024-04-09T22:02:44.936 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001918', 1: '00000000000000000000:00000000000000003888', 2: '00000000000000000000:00000000000000004460', 3: '00000000000000000000:00000000000000002622'} 2024-04-09T22:02:44.936 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T22:02:44.936 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-09T22:02:44.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:44.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:44.936 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:44.984 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:44.985 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:45.033 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:02:45.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.033 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:48:49.750310Z" 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000004593", 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:02:23.775262Z" 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:59:49.766196Z" 2024-04-09T22:02:45.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.035 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-09T22:02:45.035 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:48:49.754711Z" 2024-04-09T22:02:45.035 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.035 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:02:45.036 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000001918', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000002764'} 2024-04-09T22:02:45.036 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-09T22:02:45.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:45.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:45.037 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:45.089 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:45.089 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:45.131 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:02:45.131 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:02:45.131 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:02:45.131 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:02:45.131 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 1955642440523705218 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000001918", 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.132 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:48:49.750310Z" 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000004593", 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.133 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:02:23.775262Z" 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000004037", 2024-04-09T22:02:45.134 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:59:49.766196Z" 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-09T22:02:45.135 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:48:49.754711Z" 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 0, 2024-04-09T22:02:45.136 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 0 2024-04-09T22:02:45.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.137 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:02:45Z" 2024-04-09T22:02:45.137 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:02:45.138 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1955642440523705218 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:48:49.750310Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:02:23.775262Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004037", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:59:49.766196Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:48:49.754711Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-09T22:02:45Z" } 2024-04-09T22:02:45.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:48:49.750310Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:02:23.775262Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004037', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:59:49.766196Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:48:49.754711Z'}}] 2024-04-09T22:02:45.138 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001918', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000002764'} 2024-04-09T22:02:45.138 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001918', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004037', 3: '00000000000000000000:00000000000000002764'} 2024-04-09T22:02:45.138 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-09T22:02:45.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-04-09T22:02:45.140 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:02:45.141 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:02:45.141 DEBUG:boto:Method: PUT 2024-04-09T22:02:45.141 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:02:45.141 DEBUG:boto:Data: 2024-04-09T22:02:45.141 DEBUG:boto:Headers: {} 2024-04-09T22:02:45.141 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:45.141 DEBUG:boto:Port: 8000 2024-04-09T22:02:45.141 DEBUG:boto:Params: {} 2024-04-09T22:02:45.141 DEBUG:boto:Token: None 2024-04-09T22:02:45.142 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:02:45 GMT /xryqqk-16/ 2024-04-09T22:02:45.142 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1rIc80aSSsl063snIuGPSYI37WU= 2024-04-09T22:02:45.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1rIc80aSSsl063snIuGPSYI37WU=', 'Content-Length': '0'} 2024-04-09T22:02:45.154 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000030b52955c885aa1e-006615bb05-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:02:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:45.155 DEBUG:rgw_multi.tests:created bucket=xryqqk-16 2024-04-09T22:02:45.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:45.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:45.155 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:45.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:45.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:45.237 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:02:45.250 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:02:45.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700165.153210_266.1", 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:02:45.153210Z" 2024-04-09T22:02:45.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.252 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:02:45.252 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:02:45.252 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.252 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:02:45.252 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700165.153210_266.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:02:45.252 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:02:45.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:45.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:45.253 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:45.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:45.304 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:45.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:45.330+0000 7ff3b9cd1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:02:45.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:45.331+0000 7ff3b9cd1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:02:45.344 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.345 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700165.147630_263.1", 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:02:45.147630Z", 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.349 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:02:45Z" 2024-04-09T22:02:45.350 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:02:45.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700165.147630_263.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:02:45.147630Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:02:45Z" } 2024-04-09T22:02:45.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700165.147630_263.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:02:45.147630Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:02:45.352 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700165.153210_266.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:02:45.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700165.147630_263.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:02:45.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700165.153210_266.1 target=1_1712700165.147630_263.1 2024-04-09T22:02:50.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:50.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:02:50.353 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:02:50.413 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:50.413 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:50.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:50.449+0000 7f617dc0d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:02:50.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:50.450+0000 7f617dc0d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:02:50.464 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:02:50.464 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:02:50.464 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:02:50.465 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:02:50.465 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:02:50.465 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:02:50.465 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:50.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:50.465 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:02:50.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:50.466 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:02:50.466 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:50.466 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:50.466 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:02:50.467 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:50.467 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:02:50.467 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:50.467 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:02:50.467 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:50.468 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:50.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:50.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:50.468 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:02:50.468 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:50.469 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:50.469 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:02:50.469 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:50.469 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:50.469 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:50.470 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:02:50.470 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:50.470 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:50.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:50.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:50.471 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:02:50.471 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:50.471 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:50.471 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700165.153210_266.1", 2024-04-09T22:02:50.471 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:50.472 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:50.472 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:50.472 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:02:45.153210Z", 2024-04-09T22:02:50.472 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:50.472 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:02:50.473 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:02:50.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:02:50.474 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:02:50.474 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:02:50.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:50.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:02:50.474 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:02:50Z" 2024-04-09T22:02:50.475 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:02:50.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700165.153210_266.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:02:45.153210Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:02:50Z" } 2024-04-09T22:02:50.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700165.153210_266.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:02:45.153210Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:02:50.477 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700165.153210_266.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:02:50.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700165.153210_266.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:02:50.478 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:02:50.478 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:02:50.478 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:02:50.478 DEBUG:boto:Method: HEAD 2024-04-09T22:02:50.479 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:02:50.479 DEBUG:boto:Data: 2024-04-09T22:02:50.479 DEBUG:boto:Headers: {} 2024-04-09T22:02:50.479 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:50.479 DEBUG:boto:Port: 8000 2024-04-09T22:02:50.479 DEBUG:boto:Params: {} 2024-04-09T22:02:50.480 DEBUG:boto:Token: None 2024-04-09T22:02:50.480 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:02:50 GMT /xryqqk-16/ 2024-04-09T22:02:50.480 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3bhONIewmobdst90HHc3m7GZcEQ= 2024-04-09T22:02:50.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:02:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3bhONIewmobdst90HHc3m7GZcEQ=', 'Content-Length': '0'} 2024-04-09T22:02:50.482 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c148d326a31d1360-006615bb0a-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:50.483 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:02:50.483 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:02:50.483 DEBUG:boto:Method: PUT 2024-04-09T22:02:50.483 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:02:50.483 DEBUG:boto:Data: 2024-04-09T22:02:50.483 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:02:50.484 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:02:50.484 DEBUG:boto:Port: 8000 2024-04-09T22:02:50.484 DEBUG:boto:Params: {} 2024-04-09T22:02:50.484 DEBUG:boto:Token: None 2024-04-09T22:02:50.484 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:02:50 GMT /xryqqk-16/obj 2024-04-09T22:02:50.484 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jKBdglTDBmkE1Kz4RW/2gfUJ0cY= 2024-04-09T22:02:50.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:02:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jKBdglTDBmkE1Kz4RW/2gfUJ0cY='} 2024-04-09T22:02:50.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028aa841502e19a14-006615bb0a-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:02:50.491 DEBUG:rgw_multi.tests:created initial version id=None 2024-04-09T22:02:50.491 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-16 2024-04-09T22:02:50.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:02:50.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:02:50.492 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:02:50.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:02:50.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:02:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:50.585+0000 7f046c6d4780 1 waiting to reach incremental sync.. 2024-04-09T22:02:55.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:02:55.586+0000 7f046c6d4780 1 waiting to reach incremental sync.. 2024-04-09T22:03:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:00.587+0000 7f046c6d4780 1 bucket sync caught up with source: 2024-04-09T22:03:00.589 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.48.6, , ] 2024-04-09T22:03:00.589 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000001.48.6, , ] 2024-04-09T22:03:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:00.587+0000 7f046c6d4780 0 bucket checkpoint complete 2024-04-09T22:03:00.603 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-16 2024-04-09T22:03:00.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:00.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:00.604 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:03:00.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:00.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:00.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:00.701+0000 7f3c1c9e2780 1 bucket sync caught up with source: 2024-04-09T22:03:00.703 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.70.6, , ] 2024-04-09T22:03:00.703 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000001.70.6, , ] 2024-04-09T22:03:00.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:00.701+0000 7f3c1c9e2780 0 bucket checkpoint complete 2024-04-09T22:03:00.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-16 zones={a2, a1} 2024-04-09T22:03:00.716 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.716 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.716 DEBUG:boto:Method: HEAD 2024-04-09T22:03:00.716 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:03:00.716 DEBUG:boto:Data: 2024-04-09T22:03:00.716 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.716 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.716 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.716 DEBUG:boto:Params: {} 2024-04-09T22:03:00.717 DEBUG:boto:Token: None 2024-04-09T22:03:00.717 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/ 2024-04-09T22:03:00.717 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6QZkLyzB93PS1oyywIIRaYGrB5Q= 2024-04-09T22:03:00.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6QZkLyzB93PS1oyywIIRaYGrB5Q=', 'Content-Length': '0'} 2024-04-09T22:03:00.719 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080906db511e5a4bc-006615bb14-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.720 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.720 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.720 DEBUG:boto:Method: HEAD 2024-04-09T22:03:00.720 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:03:00.720 DEBUG:boto:Data: 2024-04-09T22:03:00.720 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.720 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.720 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.720 DEBUG:boto:Params: {} 2024-04-09T22:03:00.721 DEBUG:boto:Token: None 2024-04-09T22:03:00.721 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/ 2024-04-09T22:03:00.721 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6QZkLyzB93PS1oyywIIRaYGrB5Q= 2024-04-09T22:03:00.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6QZkLyzB93PS1oyywIIRaYGrB5Q=', 'Content-Length': '0'} 2024-04-09T22:03:00.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d900c5241f43482-006615bb14-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.723 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:03:00.723 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.723 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.724 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.724 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.724 DEBUG:boto:Method: GET 2024-04-09T22:03:00.724 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.724 DEBUG:boto:Data: 2024-04-09T22:03:00.724 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.724 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.724 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.724 DEBUG:boto:Params: {} 2024-04-09T22:03:00.725 DEBUG:boto:Token: None 2024-04-09T22:03:00.725 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/?versions 2024-04-09T22:03:00.725 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI= 2024-04-09T22:03:00.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI=', 'Content-Length': '0'} 2024-04-09T22:03:00.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009014bc5cc2646937-006615bb14-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.727 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:03:00.728 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:03:00.728 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.728 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.729 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.729 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.729 DEBUG:boto:Method: GET 2024-04-09T22:03:00.729 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.729 DEBUG:boto:Data: 2024-04-09T22:03:00.729 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.729 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.729 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.729 DEBUG:boto:Params: {} 2024-04-09T22:03:00.730 DEBUG:boto:Token: None 2024-04-09T22:03:00.730 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/?versions 2024-04-09T22:03:00.730 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI= 2024-04-09T22:03:00.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI=', 'Content-Length': '0'} 2024-04-09T22:03:00.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000477c3bd3bc87d58e-006615bb14-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.732 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:03:00.732 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.733 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.733 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.733 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.733 DEBUG:boto:Method: GET 2024-04-09T22:03:00.733 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.733 DEBUG:boto:Data: 2024-04-09T22:03:00.733 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.733 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.733 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.733 DEBUG:boto:Params: {} 2024-04-09T22:03:00.734 DEBUG:boto:Token: None 2024-04-09T22:03:00.734 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/?versions 2024-04-09T22:03:00.734 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI= 2024-04-09T22:03:00.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI=', 'Content-Length': '0'} 2024-04-09T22:03:00.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb61a83d0390d8f-006615bb14-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.778 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.779 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.779 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.779 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.779 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.780 DEBUG:boto:Method: GET 2024-04-09T22:03:00.780 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:00.780 DEBUG:boto:Data: 2024-04-09T22:03:00.780 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.780 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.780 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.781 DEBUG:boto:Params: {} 2024-04-09T22:03:00.781 DEBUG:boto:Token: None 2024-04-09T22:03:00.781 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/?versions 2024-04-09T22:03:00.781 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI= 2024-04-09T22:03:00.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pYgWyx6xEkBkmu1U7qPtZZBxVzI=', 'Content-Length': '0'} 2024-04-09T22:03:00.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba3c45e72d3477a3-006615bb14-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.784 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:00.784 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.784 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.784 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.784 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.785 DEBUG:boto:Method: GET 2024-04-09T22:03:00.785 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.785 DEBUG:boto:Data: 2024-04-09T22:03:00.785 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.785 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.785 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.785 DEBUG:boto:Params: {} 2024-04-09T22:03:00.785 DEBUG:boto:Token: None 2024-04-09T22:03:00.785 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.786 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY= 2024-04-09T22:03:00.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY=', 'Content-Length': '0'} 2024-04-09T22:03:00.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020313ec18159ccc0-006615bb14-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.788 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.788 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.788 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.788 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.788 DEBUG:boto:Method: GET 2024-04-09T22:03:00.788 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.788 DEBUG:boto:Data: 2024-04-09T22:03:00.788 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.789 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.789 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.789 DEBUG:boto:Params: {} 2024-04-09T22:03:00.789 DEBUG:boto:Token: None 2024-04-09T22:03:00.789 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.789 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY= 2024-04-09T22:03:00.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY=', 'Content-Length': '0'} 2024-04-09T22:03:00.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d8c8167b0d5adf67-006615bb14-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.792 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.792 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.793 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.793 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.793 DEBUG:boto:Method: HEAD 2024-04-09T22:03:00.793 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.793 DEBUG:boto:Data: 2024-04-09T22:03:00.793 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.793 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.793 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.793 DEBUG:boto:Params: {} 2024-04-09T22:03:00.793 DEBUG:boto:Token: None 2024-04-09T22:03:00.793 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.794 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:meOLOtZjEJ8TZESyP5+RcAeE0ns= 2024-04-09T22:03:00.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:meOLOtZjEJ8TZESyP5+RcAeE0ns=', 'Content-Length': '0'} 2024-04-09T22:03:00.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a8e3027efbfc70b-006615bb14-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.795 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.795 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.796 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.796 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.796 DEBUG:boto:Method: HEAD 2024-04-09T22:03:00.796 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.796 DEBUG:boto:Data: 2024-04-09T22:03:00.796 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.796 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.796 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.796 DEBUG:boto:Params: {} 2024-04-09T22:03:00.796 DEBUG:boto:Token: None 2024-04-09T22:03:00.797 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.797 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:meOLOtZjEJ8TZESyP5+RcAeE0ns= 2024-04-09T22:03:00.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:meOLOtZjEJ8TZESyP5+RcAeE0ns=', 'Content-Length': '0'} 2024-04-09T22:03:00.798 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097994ddcc52affed-006615bb14-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:00.798 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.799 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.799 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.799 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.801 DEBUG:boto:Method: GET 2024-04-09T22:03:00.801 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.801 DEBUG:boto:Data: 2024-04-09T22:03:00.802 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.802 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.802 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.803 DEBUG:boto:Params: {} 2024-04-09T22:03:00.803 DEBUG:boto:Token: None 2024-04-09T22:03:00.803 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.803 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY= 2024-04-09T22:03:00.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY=', 'Content-Length': '0'} 2024-04-09T22:03:00.805 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f274043773d7c64-006615bb14-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.805 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.805 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.806 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.806 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.806 DEBUG:boto:Method: GET 2024-04-09T22:03:00.806 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.806 DEBUG:boto:Data: 2024-04-09T22:03:00.806 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.806 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.807 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.807 DEBUG:boto:Params: {} 2024-04-09T22:03:00.807 DEBUG:boto:Token: None 2024-04-09T22:03:00.807 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:00.807 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY= 2024-04-09T22:03:00.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wO8ulnVt1mEonhLMLZ3rPih+dfY=', 'Content-Length': '0'} 2024-04-09T22:03:00.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000899200c17a999516-006615bb14-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.809 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.809 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.809 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.809 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.809 DEBUG:boto:Method: GET 2024-04-09T22:03:00.809 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:00.809 DEBUG:boto:Data: 2024-04-09T22:03:00.810 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.810 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.810 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.810 DEBUG:boto:Params: {} 2024-04-09T22:03:00.810 DEBUG:boto:Token: None 2024-04-09T22:03:00.810 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/ 2024-04-09T22:03:00.810 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4= 2024-04-09T22:03:00.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4=', 'Content-Length': '0'} 2024-04-09T22:03:00.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c93245f7e259808d-006615bb14-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.813 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.813 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.814 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.814 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.814 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.814 DEBUG:boto:Method: GET 2024-04-09T22:03:00.814 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:00.814 DEBUG:boto:Data: 2024-04-09T22:03:00.814 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.814 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.814 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.814 DEBUG:boto:Params: {} 2024-04-09T22:03:00.815 DEBUG:boto:Token: None 2024-04-09T22:03:00.815 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/ 2024-04-09T22:03:00.815 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4= 2024-04-09T22:03:00.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4=', 'Content-Length': '0'} 2024-04-09T22:03:00.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000373126ae1b01778-006615bb14-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.858 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.859 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.859 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.859 DEBUG:boto:Method: HEAD 2024-04-09T22:03:00.859 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:00.859 DEBUG:boto:Data: 2024-04-09T22:03:00.859 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.859 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.859 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.859 DEBUG:boto:Params: {} 2024-04-09T22:03:00.860 DEBUG:boto:Token: None 2024-04-09T22:03:00.860 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj 2024-04-09T22:03:00.860 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sC0PN6YC0KGyZxSELi0Fq+8SI+E= 2024-04-09T22:03:00.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sC0PN6YC0KGyZxSELi0Fq+8SI+E=', 'Content-Length': '0'} 2024-04-09T22:03:00.861 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002eed957aa42b9857-006615bb14-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.862 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.862 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.862 DEBUG:boto:Method: HEAD 2024-04-09T22:03:00.862 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:00.862 DEBUG:boto:Data: 2024-04-09T22:03:00.862 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.862 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.862 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.862 DEBUG:boto:Params: {} 2024-04-09T22:03:00.862 DEBUG:boto:Token: None 2024-04-09T22:03:00.863 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj 2024-04-09T22:03:00.863 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sC0PN6YC0KGyZxSELi0Fq+8SI+E= 2024-04-09T22:03:00.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sC0PN6YC0KGyZxSELi0Fq+8SI+E=', 'Content-Length': '0'} 2024-04-09T22:03:00.864 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005f1b06d6d251674c-006615bb14-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:00.865 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.865 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.865 DEBUG:boto:Method: GET 2024-04-09T22:03:00.865 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:00.865 DEBUG:boto:Data: 2024-04-09T22:03:00.865 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.866 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.866 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.866 DEBUG:boto:Params: {} 2024-04-09T22:03:00.866 DEBUG:boto:Token: None 2024-04-09T22:03:00.866 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj 2024-04-09T22:03:00.866 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aJMT5meXvH6mnO87vAnoxwyWew0= 2024-04-09T22:03:00.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aJMT5meXvH6mnO87vAnoxwyWew0=', 'Content-Length': '0'} 2024-04-09T22:03:00.868 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2fa182896aa6991-006615bb14-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.868 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:00.868 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:00.868 DEBUG:boto:Method: GET 2024-04-09T22:03:00.868 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:00.868 DEBUG:boto:Data: 2024-04-09T22:03:00.868 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.869 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.869 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.869 DEBUG:boto:Params: {} 2024-04-09T22:03:00.869 DEBUG:boto:Token: None 2024-04-09T22:03:00.869 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/obj 2024-04-09T22:03:00.869 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aJMT5meXvH6mnO87vAnoxwyWew0= 2024-04-09T22:03:00.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aJMT5meXvH6mnO87vAnoxwyWew0=', 'Content-Length': '0'} 2024-04-09T22:03:00.871 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e53ce2d96bb29ca5-006615bb14-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.871 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.871 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.871 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.871 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.871 DEBUG:boto:Method: GET 2024-04-09T22:03:00.871 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:00.871 DEBUG:boto:Data: 2024-04-09T22:03:00.872 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.872 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:00.872 DEBUG:boto:Port: 8001 2024-04-09T22:03:00.872 DEBUG:boto:Params: {} 2024-04-09T22:03:00.872 DEBUG:boto:Token: None 2024-04-09T22:03:00.872 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/ 2024-04-09T22:03:00.872 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4= 2024-04-09T22:03:00.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4=', 'Content-Length': '0'} 2024-04-09T22:03:00.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001432732527818e0f-006615bb14-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.874 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.875 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.875 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.875 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.875 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:00.875 DEBUG:boto:Method: GET 2024-04-09T22:03:00.875 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:00.875 DEBUG:boto:Data: 2024-04-09T22:03:00.875 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.876 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.876 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.876 DEBUG:boto:Params: {} 2024-04-09T22:03:00.876 DEBUG:boto:Token: None 2024-04-09T22:03:00.876 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/ 2024-04-09T22:03:00.876 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4= 2024-04-09T22:03:00.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3nNnTzBbsKGXa6xMzbAbsBB7Du4=', 'Content-Length': '0'} 2024-04-09T22:03:00.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81ff31eb2a84014-006615bb14-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.879 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:00.879 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-16 zones={a2, a1} 2024-04-09T22:03:00.879 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:00.879 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:00.879 DEBUG:boto:path=/xryqqk-16/?versioning 2024-04-09T22:03:00.879 DEBUG:boto:auth_path=/xryqqk-16/?versioning 2024-04-09T22:03:00.879 DEBUG:boto:Method: PUT 2024-04-09T22:03:00.879 DEBUG:boto:Path: /xryqqk-16/?versioning 2024-04-09T22:03:00.880 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:03:00.880 DEBUG:boto:Headers: {} 2024-04-09T22:03:00.880 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:00.880 DEBUG:boto:Port: 8000 2024-04-09T22:03:00.880 DEBUG:boto:Params: {} 2024-04-09T22:03:00.880 DEBUG:boto:Token: None 2024-04-09T22:03:00.880 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:03:00 GMT /xryqqk-16/?versioning 2024-04-09T22:03:00.880 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:btlAzfsfg86bP3YN9jQf1axef4s= 2024-04-09T22:03:00.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:00 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:btlAzfsfg86bP3YN9jQf1axef4s=', 'Content-Length': '228'} 2024-04-09T22:03:00.885 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aa6a7b373fd77ee3-006615bb14-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:00.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:00.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:00.885 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:03:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:00.964 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:03:00.976 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:03:00.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:00.976 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:03:00.976 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700180.884583_268.1", 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:03:00.884583Z" 2024-04-09T22:03:00.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:00.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:00.978 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:03:00.978 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:03:00.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:00.978 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:03:00.979 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700180.884583_268.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:00.979 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:03:00.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:00.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:00.980 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:03:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:01.030 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:01.059+0000 7f495f3b0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:03:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:01.060+0000 7f495f3b0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:03:01.075 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:03:01.075 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:03:01.075 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:03:01.075 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:01.076 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:01.077 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:03:01.078 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700180.881993_267.1", 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:03:00.881993Z", 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:01.079 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:01.080 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:03:01.081 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:03:01.081 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:03:01.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:01.081 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:03:01Z" 2024-04-09T22:03:01.081 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:03:01.082 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700180.881993_267.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:03:00.881993Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:03:01Z" } 2024-04-09T22:03:01.082 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700180.881993_267.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:03:00.881993Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:03:01.082 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700180.884583_268.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:01.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700180.881993_267.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:01.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700180.884583_268.1 target=1_1712700180.881993_267.1 2024-04-09T22:03:06.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:06.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:06.084 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:03:06.141 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:06.141 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:06.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:06.174+0000 7f3d2e04b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:03:06.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:06.175+0000 7f3d2e04b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:03:06.188 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:03:06.188 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:03:06.188 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:03:06.188 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:03:06.188 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:06.189 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:03:06.190 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:06.191 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700180.884583_268.1", 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:03:00.884583Z", 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:06.192 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:03:06.193 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:03:06.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:06.194 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:03:06Z" 2024-04-09T22:03:06.194 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:03:06.195 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700180.884583_268.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:03:00.884583Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:03:06Z" } 2024-04-09T22:03:06.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700180.884583_268.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:03:00.884583Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:03:06.195 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700180.884583_268.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:06.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700180.884583_268.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:06.195 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:03:06.195 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:06.196 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:06.196 DEBUG:boto:Method: HEAD 2024-04-09T22:03:06.196 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:03:06.196 DEBUG:boto:Data: 2024-04-09T22:03:06.196 DEBUG:boto:Headers: {} 2024-04-09T22:03:06.196 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:06.196 DEBUG:boto:Port: 8000 2024-04-09T22:03:06.196 DEBUG:boto:Params: {} 2024-04-09T22:03:06.196 DEBUG:boto:Token: None 2024-04-09T22:03:06.197 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:06 GMT /xryqqk-16/ 2024-04-09T22:03:06.197 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qTIm09GpG4vzxObpI3gb9vbQu+E= 2024-04-09T22:03:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:06 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qTIm09GpG4vzxObpI3gb9vbQu+E=', 'Content-Length': '0'} 2024-04-09T22:03:06.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0e61bece2d222be-006615bb1a-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:06.199 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:06.199 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:06.199 DEBUG:boto:Method: PUT 2024-04-09T22:03:06.200 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:06.200 DEBUG:boto:Data: 2024-04-09T22:03:06.200 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:03:06.200 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:06.200 DEBUG:boto:Port: 8000 2024-04-09T22:03:06.200 DEBUG:boto:Params: {} 2024-04-09T22:03:06.200 DEBUG:boto:Token: None 2024-04-09T22:03:06.200 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:03:06 GMT /xryqqk-16/obj 2024-04-09T22:03:06.200 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zNSlJ1z1Aw7n4V44UWedGAWH2iM= 2024-04-09T22:03:06.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:03:06 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zNSlJ1z1Aw7n4V44UWedGAWH2iM='} 2024-04-09T22:03:06.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-amz-request-id', 'tx00000cfe28c3e0543a9a5-006615bb1a-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:06.209 DEBUG:rgw_multi.tests:created new version id=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:06.209 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-16 2024-04-09T22:03:06.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:06.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:06.210 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:03:06.267 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:06.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:06.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:06.302+0000 7f8f15dc2780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:06.302 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.48.6, , ] 2024-04-09T22:03:06.302 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:11.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:11.304+0000 7f8f15dc2780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:11.304 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.48.6, , ] 2024-04-09T22:03:11.304 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:16.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:16.304+0000 7f8f15dc2780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:16.306 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.48.6, , ] 2024-04-09T22:03:16.306 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:21.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:21.305+0000 7f8f15dc2780 1 bucket sync caught up with source: 2024-04-09T22:03:21.307 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:21.307 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:21.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:21.305+0000 7f8f15dc2780 0 bucket checkpoint complete 2024-04-09T22:03:21.324 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-16 2024-04-09T22:03:21.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:21.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:21.324 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:03:21.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:21.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:21.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:21.425+0000 7f1e41ea0780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:21.426 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.70.6, , ] 2024-04-09T22:03:21.427 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:03:26.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:26.426+0000 7f1e41ea0780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:26.428 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.70.6, , ] 2024-04-09T22:03:26.428 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:03:31.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:31.428+0000 7f1e41ea0780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:31.430 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.70.6, , ] 2024-04-09T22:03:31.430 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:03:36.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:36.429+0000 7f1e41ea0780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:36.431 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.70.6, , ] 2024-04-09T22:03:36.431 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:03:41.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:41.430+0000 7f1e41ea0780 1 bucket sync caught up with source: 2024-04-09T22:03:41.432 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:03:41.432 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:03:41.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:41.430+0000 7f1e41ea0780 0 bucket checkpoint complete 2024-04-09T22:03:41.449 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-16 zones={a2, a1} 2024-04-09T22:03:41.449 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.449 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.449 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.450 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:03:41.450 DEBUG:boto:Data: 2024-04-09T22:03:41.450 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.450 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.450 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.450 DEBUG:boto:Params: {} 2024-04-09T22:03:41.450 DEBUG:boto:Token: None 2024-04-09T22:03:41.451 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.451 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k9fvDqvtrmej6hWQaOQ7M9TI9us= 2024-04-09T22:03:41.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k9fvDqvtrmej6hWQaOQ7M9TI9us=', 'Content-Length': '0'} 2024-04-09T22:03:41.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000956dc5475db101fc-006615bb3d-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.455 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.455 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.455 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.455 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:03:41.455 DEBUG:boto:Data: 2024-04-09T22:03:41.455 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.455 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.455 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.455 DEBUG:boto:Params: {} 2024-04-09T22:03:41.457 DEBUG:boto:Token: None 2024-04-09T22:03:41.457 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.457 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k9fvDqvtrmej6hWQaOQ7M9TI9us= 2024-04-09T22:03:41.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k9fvDqvtrmej6hWQaOQ7M9TI9us=', 'Content-Length': '0'} 2024-04-09T22:03:41.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004422d95d074eb85e-006615bb3d-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:03:41.460 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.460 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.460 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.460 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.460 DEBUG:boto:Method: GET 2024-04-09T22:03:41.460 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.460 DEBUG:boto:Data: 2024-04-09T22:03:41.460 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.461 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.461 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.461 DEBUG:boto:Params: {} 2024-04-09T22:03:41.461 DEBUG:boto:Token: None 2024-04-09T22:03:41.461 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/?versions 2024-04-09T22:03:41.461 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs= 2024-04-09T22:03:41.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs=', 'Content-Length': '0'} 2024-04-09T22:03:41.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167a5f489e21fd29-006615bb3d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.464 DEBUG:boto:b'xryqqk-16999falseobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFtrue2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:03:41.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:03:41.465 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:03:41.465 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.465 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.465 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.465 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.465 DEBUG:boto:Method: GET 2024-04-09T22:03:41.465 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.465 DEBUG:boto:Data: 2024-04-09T22:03:41.465 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.466 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.466 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.466 DEBUG:boto:Params: {} 2024-04-09T22:03:41.466 DEBUG:boto:Token: None 2024-04-09T22:03:41.466 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/?versions 2024-04-09T22:03:41.466 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs= 2024-04-09T22:03:41.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs=', 'Content-Length': '0'} 2024-04-09T22:03:41.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000995c1b4df69b3765-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.469 DEBUG:boto:b'xryqqk-16999falseobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFtrue2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:03:41.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:03:41.469 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.469 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.469 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.470 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.470 DEBUG:boto:Method: GET 2024-04-09T22:03:41.470 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.470 DEBUG:boto:Data: 2024-04-09T22:03:41.470 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.470 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.470 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.470 DEBUG:boto:Params: {} 2024-04-09T22:03:41.470 DEBUG:boto:Token: None 2024-04-09T22:03:41.470 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/?versions 2024-04-09T22:03:41.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs= 2024-04-09T22:03:41.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs=', 'Content-Length': '0'} 2024-04-09T22:03:41.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5f459850273a6dc-006615bb3d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.514 DEBUG:boto:b'xryqqk-16999falseobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFtrue2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.515 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.515 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.515 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.515 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.515 DEBUG:boto:Method: GET 2024-04-09T22:03:41.515 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:03:41.516 DEBUG:boto:Data: 2024-04-09T22:03:41.516 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.516 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.516 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.516 DEBUG:boto:Params: {} 2024-04-09T22:03:41.516 DEBUG:boto:Token: None 2024-04-09T22:03:41.516 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/?versions 2024-04-09T22:03:41.516 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs= 2024-04-09T22:03:41.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OPNq73l2KM8GJuuhOK1WjrmpuWs=', 'Content-Length': '0'} 2024-04-09T22:03:41.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5e9df5dc2ea72f5-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.519 DEBUG:boto:b'xryqqk-16999falseobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFtrue2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-09T22:02:50.486Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:41.520 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.520 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.520 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.520 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.520 DEBUG:boto:Method: GET 2024-04-09T22:03:41.521 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.521 DEBUG:boto:Data: 2024-04-09T22:03:41.521 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.521 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.521 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.521 DEBUG:boto:Params: {} 2024-04-09T22:03:41.521 DEBUG:boto:Token: None 2024-04-09T22:03:41.521 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.522 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001170a0d32cdac987-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.524 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.524 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.524 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.525 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.525 DEBUG:boto:Method: GET 2024-04-09T22:03:41.525 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.525 DEBUG:boto:Data: 2024-04-09T22:03:41.525 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.525 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.525 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.525 DEBUG:boto:Params: {} 2024-04-09T22:03:41.525 DEBUG:boto:Token: None 2024-04-09T22:03:41.526 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.526 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c3b33ef54b0dfed-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.528 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.528 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.528 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.528 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.528 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.528 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.528 DEBUG:boto:Data: 2024-04-09T22:03:41.528 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.528 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.529 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.529 DEBUG:boto:Params: {} 2024-04-09T22:03:41.529 DEBUG:boto:Token: None 2024-04-09T22:03:41.529 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.529 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ceRyA8MS+HobKqk6oMgiUHztVcI= 2024-04-09T22:03:41.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ceRyA8MS+HobKqk6oMgiUHztVcI=', 'Content-Length': '0'} 2024-04-09T22:03:41.531 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cd8046900b3b2a5-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.531 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.531 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.531 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.531 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.531 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.531 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.531 DEBUG:boto:Data: 2024-04-09T22:03:41.532 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.532 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.532 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.532 DEBUG:boto:Params: {} 2024-04-09T22:03:41.532 DEBUG:boto:Token: None 2024-04-09T22:03:41.532 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ceRyA8MS+HobKqk6oMgiUHztVcI= 2024-04-09T22:03:41.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ceRyA8MS+HobKqk6oMgiUHztVcI=', 'Content-Length': '0'} 2024-04-09T22:03:41.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c910361b6a63152d-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:41.534 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.535 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.535 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.535 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.535 DEBUG:boto:Method: GET 2024-04-09T22:03:41.535 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.535 DEBUG:boto:Data: 2024-04-09T22:03:41.535 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.535 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.535 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.535 DEBUG:boto:Params: {} 2024-04-09T22:03:41.536 DEBUG:boto:Token: None 2024-04-09T22:03:41.536 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.536 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.537 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008cc237b192a649bb-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.538 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.538 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.538 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.538 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.538 DEBUG:boto:Method: GET 2024-04-09T22:03:41.538 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.538 DEBUG:boto:Data: 2024-04-09T22:03:41.538 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.541 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.541 DEBUG:boto:Params: {} 2024-04-09T22:03:41.541 DEBUG:boto:Token: None 2024-04-09T22:03:41.542 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.542 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e68c6d8100a468e8-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.544 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.545 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.545 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.545 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.545 DEBUG:boto:Method: GET 2024-04-09T22:03:41.545 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.545 DEBUG:boto:Data: 2024-04-09T22:03:41.545 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.545 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.545 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.545 DEBUG:boto:Params: {} 2024-04-09T22:03:41.545 DEBUG:boto:Token: None 2024-04-09T22:03:41.546 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.546 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b81e230824a64401-006615bb3d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.590 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.591 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.591 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.591 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.591 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.591 DEBUG:boto:Method: GET 2024-04-09T22:03:41.591 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.591 DEBUG:boto:Data: 2024-04-09T22:03:41.592 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.592 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.592 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.592 DEBUG:boto:Params: {} 2024-04-09T22:03:41.592 DEBUG:boto:Token: None 2024-04-09T22:03:41.592 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f39f69b0ff2dec31-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.595 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.595 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.595 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.596 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.596 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:41.596 DEBUG:boto:Data: 2024-04-09T22:03:41.596 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.596 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.596 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.596 DEBUG:boto:Params: {} 2024-04-09T22:03:41.596 DEBUG:boto:Token: None 2024-04-09T22:03:41.596 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj 2024-04-09T22:03:41.597 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4= 2024-04-09T22:03:41.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4=', 'Content-Length': '0'} 2024-04-09T22:03:41.599 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000375757beb126d646-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.599 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.599 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.599 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.599 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:41.599 DEBUG:boto:Data: 2024-04-09T22:03:41.599 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.599 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.600 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.600 DEBUG:boto:Params: {} 2024-04-09T22:03:41.600 DEBUG:boto:Token: None 2024-04-09T22:03:41.600 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj 2024-04-09T22:03:41.600 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4= 2024-04-09T22:03:41.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4=', 'Content-Length': '0'} 2024-04-09T22:03:41.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068bf730a3f371293-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:41.602 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.603 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.603 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.603 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.603 DEBUG:boto:Method: GET 2024-04-09T22:03:41.603 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.603 DEBUG:boto:Data: 2024-04-09T22:03:41.603 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.603 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.603 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.603 DEBUG:boto:Params: {} 2024-04-09T22:03:41.603 DEBUG:boto:Token: None 2024-04-09T22:03:41.604 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.604 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.605 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006bd872406f09bf02-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.605 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.606 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.606 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.606 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.606 DEBUG:boto:Method: GET 2024-04-09T22:03:41.606 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.606 DEBUG:boto:Data: 2024-04-09T22:03:41.606 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.606 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.606 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.606 DEBUG:boto:Params: {} 2024-04-09T22:03:41.607 DEBUG:boto:Token: None 2024-04-09T22:03:41.607 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.607 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005e9fe0ddaa4a01d4-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.609 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.609 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.609 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.609 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.609 DEBUG:boto:Method: GET 2024-04-09T22:03:41.609 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.609 DEBUG:boto:Data: 2024-04-09T22:03:41.609 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.609 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.610 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.610 DEBUG:boto:Params: {} 2024-04-09T22:03:41.610 DEBUG:boto:Token: None 2024-04-09T22:03:41.610 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.610 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028c24e27b6297b05-006615bb3d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.613 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.613 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.613 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.613 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.613 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.614 DEBUG:boto:Method: GET 2024-04-09T22:03:41.614 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.614 DEBUG:boto:Data: 2024-04-09T22:03:41.614 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.614 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.614 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.614 DEBUG:boto:Params: {} 2024-04-09T22:03:41.614 DEBUG:boto:Token: None 2024-04-09T22:03:41.614 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.614 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092ee831b060b190f-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.617 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:41.617 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.617 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.617 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.618 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.618 DEBUG:boto:Method: GET 2024-04-09T22:03:41.618 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.618 DEBUG:boto:Data: 2024-04-09T22:03:41.618 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.618 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.618 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.618 DEBUG:boto:Params: {} 2024-04-09T22:03:41.618 DEBUG:boto:Token: None 2024-04-09T22:03:41.618 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.619 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4= 2024-04-09T22:03:41.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4=', 'Content-Length': '0'} 2024-04-09T22:03:41.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d062f92ee2227ad3-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.620 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.621 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.621 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.621 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.621 DEBUG:boto:Method: GET 2024-04-09T22:03:41.621 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.621 DEBUG:boto:Data: 2024-04-09T22:03:41.621 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.621 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.621 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.621 DEBUG:boto:Params: {} 2024-04-09T22:03:41.622 DEBUG:boto:Token: None 2024-04-09T22:03:41.622 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.622 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4= 2024-04-09T22:03:41.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4=', 'Content-Length': '0'} 2024-04-09T22:03:41.624 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000068c23ebf2e4393e8-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.624 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.625 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.625 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.625 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.625 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.625 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.625 DEBUG:boto:Data: 2024-04-09T22:03:41.625 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.625 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.625 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.625 DEBUG:boto:Params: {} 2024-04-09T22:03:41.625 DEBUG:boto:Token: None 2024-04-09T22:03:41.626 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.626 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LMf+m+chTE1xZA8s1lF785kRdz0= 2024-04-09T22:03:41.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LMf+m+chTE1xZA8s1lF785kRdz0=', 'Content-Length': '0'} 2024-04-09T22:03:41.628 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a4060cf402634ff7-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.628 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.628 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.628 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.628 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.628 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.628 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.628 DEBUG:boto:Data: 2024-04-09T22:03:41.628 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.629 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.629 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.629 DEBUG:boto:Params: {} 2024-04-09T22:03:41.629 DEBUG:boto:Token: None 2024-04-09T22:03:41.629 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.629 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LMf+m+chTE1xZA8s1lF785kRdz0= 2024-04-09T22:03:41.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LMf+m+chTE1xZA8s1lF785kRdz0=', 'Content-Length': '0'} 2024-04-09T22:03:41.631 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc1651f50a67820f-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:41.631 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.631 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.631 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.631 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.631 DEBUG:boto:Method: GET 2024-04-09T22:03:41.632 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.632 DEBUG:boto:Data: 2024-04-09T22:03:41.632 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.632 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.632 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.632 DEBUG:boto:Params: {} 2024-04-09T22:03:41.632 DEBUG:boto:Token: None 2024-04-09T22:03:41.633 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.633 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4= 2024-04-09T22:03:41.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4=', 'Content-Length': '0'} 2024-04-09T22:03:41.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f3afb399d9b5849c-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.635 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.635 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.635 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.635 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.635 DEBUG:boto:Method: GET 2024-04-09T22:03:41.635 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.635 DEBUG:boto:Data: 2024-04-09T22:03:41.635 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.635 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.635 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.636 DEBUG:boto:Params: {} 2024-04-09T22:03:41.636 DEBUG:boto:Token: None 2024-04-09T22:03:41.636 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:03:41.636 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4= 2024-04-09T22:03:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Qc7GPQozFJRR+5Bo+aYjTtOUbi4=', 'Content-Length': '0'} 2024-04-09T22:03:41.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:02:50 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007e4137eade7934ad-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.638 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.638 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.638 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.638 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.638 DEBUG:boto:Method: GET 2024-04-09T22:03:41.638 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.639 DEBUG:boto:Data: 2024-04-09T22:03:41.639 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.639 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.639 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.639 DEBUG:boto:Params: {} 2024-04-09T22:03:41.639 DEBUG:boto:Token: None 2024-04-09T22:03:41.639 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.639 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860b645f13a9f57f-006615bb3d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.682 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.683 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.683 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.683 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.683 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.684 DEBUG:boto:Method: GET 2024-04-09T22:03:41.684 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.684 DEBUG:boto:Data: 2024-04-09T22:03:41.684 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.684 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.684 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.684 DEBUG:boto:Params: {} 2024-04-09T22:03:41.685 DEBUG:boto:Token: None 2024-04-09T22:03:41.685 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.685 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081613be127c1c24a-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.688 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.689 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.689 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.689 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.689 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:41.689 DEBUG:boto:Data: 2024-04-09T22:03:41.689 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.689 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.689 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.689 DEBUG:boto:Params: {} 2024-04-09T22:03:41.689 DEBUG:boto:Token: None 2024-04-09T22:03:41.690 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj 2024-04-09T22:03:41.690 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4= 2024-04-09T22:03:41.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4=', 'Content-Length': '0'} 2024-04-09T22:03:41.692 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbc9d070ae03338a-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.692 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.692 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.692 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.692 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:41.693 DEBUG:boto:Data: 2024-04-09T22:03:41.693 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.693 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.693 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.693 DEBUG:boto:Params: {} 2024-04-09T22:03:41.693 DEBUG:boto:Token: None 2024-04-09T22:03:41.693 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj 2024-04-09T22:03:41.693 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4= 2024-04-09T22:03:41.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TThojcVmXg3CaN9BFNrXKrdpCp4=', 'Content-Length': '0'} 2024-04-09T22:03:41.695 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa99e7de92269dff-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:03:41.696 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.696 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.696 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.696 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.696 DEBUG:boto:Method: GET 2024-04-09T22:03:41.696 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.696 DEBUG:boto:Data: 2024-04-09T22:03:41.696 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.696 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.696 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.696 DEBUG:boto:Params: {} 2024-04-09T22:03:41.697 DEBUG:boto:Token: None 2024-04-09T22:03:41.697 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.697 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000097ce531af62d3e0f-006615bb3d-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.699 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.699 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.699 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.699 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.699 DEBUG:boto:Method: GET 2024-04-09T22:03:41.699 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.699 DEBUG:boto:Data: 2024-04-09T22:03:41.699 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.699 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.699 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.699 DEBUG:boto:Params: {} 2024-04-09T22:03:41.700 DEBUG:boto:Token: None 2024-04-09T22:03:41.700 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:03:41.700 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY= 2024-04-09T22:03:41.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ux6+j7UhDKFHRJxhhon5uvPXxFY=', 'Content-Length': '0'} 2024-04-09T22:03:41.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a69c5eb22dac342-006615bb3d-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.702 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.702 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.702 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.702 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.702 DEBUG:boto:Method: GET 2024-04-09T22:03:41.702 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.702 DEBUG:boto:Data: 2024-04-09T22:03:41.702 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.703 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:03:41.703 DEBUG:boto:Port: 8001 2024-04-09T22:03:41.703 DEBUG:boto:Params: {} 2024-04-09T22:03:41.703 DEBUG:boto:Token: None 2024-04-09T22:03:41.703 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.703 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d85b6c05d2b7ba26-006615bb3d-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.705 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.706 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.706 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.706 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.706 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:03:41.706 DEBUG:boto:Method: GET 2024-04-09T22:03:41.706 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:03:41.706 DEBUG:boto:Data: 2024-04-09T22:03:41.706 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.707 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.707 DEBUG:boto:Params: {} 2024-04-09T22:03:41.707 DEBUG:boto:Token: None 2024-04-09T22:03:41.707 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.707 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI= 2024-04-09T22:03:41.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2c2aTHfsS/ViyXtQ6kMeaxBVkeI=', 'Content-Length': '0'} 2024-04-09T22:03:41.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d3e1cc03ce6af3-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.754 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:03:41.755 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-16 zones={a2, a1} 2024-04-09T22:03:41.755 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.755 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.755 DEBUG:boto:path=/xryqqk-16/?versioning 2024-04-09T22:03:41.755 DEBUG:boto:auth_path=/xryqqk-16/?versioning 2024-04-09T22:03:41.755 DEBUG:boto:Method: PUT 2024-04-09T22:03:41.756 DEBUG:boto:Path: /xryqqk-16/?versioning 2024-04-09T22:03:41.756 DEBUG:boto:Data: Suspended Disabled 2024-04-09T22:03:41.756 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.756 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.756 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.756 DEBUG:boto:Params: {} 2024-04-09T22:03:41.756 DEBUG:boto:Token: None 2024-04-09T22:03:41.756 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/?versioning 2024-04-09T22:03:41.756 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:NicIik56zvkhruVhnwqOFZEfubk= 2024-04-09T22:03:41.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NicIik56zvkhruVhnwqOFZEfubk=', 'Content-Length': '230'} 2024-04-09T22:03:41.762 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001df43db4a42dd5b2-006615bb3d-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:41.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:41.764 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:03:41.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:41.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:41.851 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:03:41.865 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700221.761574_277.1", 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:03:41.761574Z" 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:41.866 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:03:41.867 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700221.761574_277.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:41.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:03:41.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:41.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:03:41.868 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:03:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:41.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:41.946+0000 7fcd524fc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:03:41.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:41.947+0000 7fcd524fc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:03:41.960 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:03:41.960 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:03:41.960 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:03:41.961 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:03:41.962 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:41.963 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700221.761574_277.1", 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:03:41.761574Z", 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:03:41.964 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.965 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:03:41.966 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:03:41.966 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:03:41.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:03:41.966 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:03:41Z" 2024-04-09T22:03:41.966 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:03:41.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700221.761574_277.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:03:41.761574Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:03:41Z" } 2024-04-09T22:03:41.967 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700221.761574_277.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:03:41.761574Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:03:41.967 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700221.761574_277.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:41.967 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700221.761574_277.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:03:41.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:03:41.967 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:03:41.968 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:03:41.968 DEBUG:boto:Method: HEAD 2024-04-09T22:03:41.968 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:03:41.968 DEBUG:boto:Data: 2024-04-09T22:03:41.968 DEBUG:boto:Headers: {} 2024-04-09T22:03:41.968 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.968 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.968 DEBUG:boto:Params: {} 2024-04-09T22:03:41.968 DEBUG:boto:Token: None 2024-04-09T22:03:41.968 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/ 2024-04-09T22:03:41.969 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k9fvDqvtrmej6hWQaOQ7M9TI9us= 2024-04-09T22:03:41.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k9fvDqvtrmej6hWQaOQ7M9TI9us=', 'Content-Length': '0'} 2024-04-09T22:03:41.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f61b8664b9be56c6-006615bb3d-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.971 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:03:41.971 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:03:41.971 DEBUG:boto:Method: PUT 2024-04-09T22:03:41.972 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:03:41.972 DEBUG:boto:Data: 2024-04-09T22:03:41.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:03:41.972 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:03:41.972 DEBUG:boto:Port: 8000 2024-04-09T22:03:41.972 DEBUG:boto:Params: {} 2024-04-09T22:03:41.972 DEBUG:boto:Token: None 2024-04-09T22:03:41.972 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:03:41 GMT /xryqqk-16/obj 2024-04-09T22:03:41.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jYWu3ihWFlpcVwH4S/PMA/AwxOI= 2024-04-09T22:03:41.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:03:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jYWu3ihWFlpcVwH4S/PMA/AwxOI='} 2024-04-09T22:03:41.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000525fb571c985240b-006615bb3d-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:03:41.983 DEBUG:rgw_multi.tests:created null version id=None 2024-04-09T22:03:41.983 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-16 2024-04-09T22:03:41.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:41.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:03:41.984 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:03:42.041 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:03:42.041 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:03:42.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:42.076+0000 7efffec84780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:42.077 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:42.077 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.55.18, , ] 2024-04-09T22:03:47.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:47.077+0000 7efffec84780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:47.079 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:47.079 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.55.18, , ] 2024-04-09T22:03:52.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:52.078+0000 7efffec84780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:52.080 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:52.080 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.55.18, , ] 2024-04-09T22:03:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:03:57.079+0000 7efffec84780 1 waiting for incremental sync to catch up: 2024-04-09T22:03:57.081 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.51.13, , ] 2024-04-09T22:03:57.081 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.55.18, , ] 2024-04-09T22:04:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:02.081+0000 7efffec84780 1 bucket sync caught up with source: 2024-04-09T22:04:02.082 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000005.55.18, , ] 2024-04-09T22:04:02.082 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.55.18, , ] 2024-04-09T22:04:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:02.081+0000 7efffec84780 0 bucket checkpoint complete 2024-04-09T22:04:02.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-16 2024-04-09T22:04:02.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:04:02.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:04:02.099 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:04:02.162 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:04:02.162 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:04:02.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:02.200+0000 7f42589f3780 1 waiting for incremental sync to catch up: 2024-04-09T22:04:02.201 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:04:02.201 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.81.18, , ] 2024-04-09T22:04:07.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:07.201+0000 7f42589f3780 1 waiting for incremental sync to catch up: 2024-04-09T22:04:07.203 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:04:07.203 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.81.18, , ] 2024-04-09T22:04:12.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:12.202+0000 7f42589f3780 1 waiting for incremental sync to catch up: 2024-04-09T22:04:12.204 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:04:12.204 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.81.18, , ] 2024-04-09T22:04:17.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:17.203+0000 7f42589f3780 1 waiting for incremental sync to catch up: 2024-04-09T22:04:17.205 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.75.13, , ] 2024-04-09T22:04:17.205 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.81.18, , ] 2024-04-09T22:04:22.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:22.205+0000 7f42589f3780 1 bucket sync caught up with source: 2024-04-09T22:04:22.206 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000005.81.18, , ] 2024-04-09T22:04:22.207 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000005.81.18, , ] 2024-04-09T22:04:22.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:22.205+0000 7f42589f3780 0 bucket checkpoint complete 2024-04-09T22:04:22.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-16 zones={a2, a1} 2024-04-09T22:04:22.224 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.224 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.224 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.225 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:04:22.225 DEBUG:boto:Data: 2024-04-09T22:04:22.225 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.225 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.225 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.226 DEBUG:boto:Params: {} 2024-04-09T22:04:22.226 DEBUG:boto:Token: None 2024-04-09T22:04:22.227 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.227 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oVKjfe/L6IRYpP/yNJ7SjpAvGoU= 2024-04-09T22:04:22.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oVKjfe/L6IRYpP/yNJ7SjpAvGoU=', 'Content-Length': '0'} 2024-04-09T22:04:22.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9d8bbdf00cc26de-006615bb66-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.230 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.231 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.231 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.231 DEBUG:boto:Path: /xryqqk-16/ 2024-04-09T22:04:22.231 DEBUG:boto:Data: 2024-04-09T22:04:22.231 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.231 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.231 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.231 DEBUG:boto:Params: {} 2024-04-09T22:04:22.231 DEBUG:boto:Token: None 2024-04-09T22:04:22.232 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.232 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oVKjfe/L6IRYpP/yNJ7SjpAvGoU= 2024-04-09T22:04:22.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oVKjfe/L6IRYpP/yNJ7SjpAvGoU=', 'Content-Length': '0'} 2024-04-09T22:04:22.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e340f462edd8c735-006615bb66-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:04:22.235 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.235 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.235 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.235 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.235 DEBUG:boto:Method: GET 2024-04-09T22:04:22.235 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.235 DEBUG:boto:Data: 2024-04-09T22:04:22.235 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.235 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.235 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.235 DEBUG:boto:Params: {} 2024-04-09T22:04:22.236 DEBUG:boto:Token: None 2024-04-09T22:04:22.236 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/?versions 2024-04-09T22:04:22.236 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM= 2024-04-09T22:04:22.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM=', 'Content-Length': '0'} 2024-04-09T22:04:22.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c6b2a616daa7312-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.238 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFfalse2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:04:22.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:04:22.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:04:22.239 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.240 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.240 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.240 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.240 DEBUG:boto:Method: GET 2024-04-09T22:04:22.240 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.240 DEBUG:boto:Data: 2024-04-09T22:04:22.240 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.240 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.240 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.240 DEBUG:boto:Params: {} 2024-04-09T22:04:22.240 DEBUG:boto:Token: None 2024-04-09T22:04:22.241 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/?versions 2024-04-09T22:04:22.241 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM= 2024-04-09T22:04:22.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM=', 'Content-Length': '0'} 2024-04-09T22:04:22.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002748cded9c6d7fb-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.243 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFfalse2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:04:22.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:04:22.244 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.244 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.244 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.244 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.244 DEBUG:boto:Method: GET 2024-04-09T22:04:22.244 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.244 DEBUG:boto:Data: 2024-04-09T22:04:22.244 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.244 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.245 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.245 DEBUG:boto:Params: {} 2024-04-09T22:04:22.245 DEBUG:boto:Token: None 2024-04-09T22:04:22.245 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/?versions 2024-04-09T22:04:22.245 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM= 2024-04-09T22:04:22.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM=', 'Content-Length': '0'} 2024-04-09T22:04:22.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009476abbba430fa86-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.294 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFfalse2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.295 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.295 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.295 DEBUG:boto:path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.295 DEBUG:boto:auth_path=/xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.295 DEBUG:boto:Method: GET 2024-04-09T22:04:22.296 DEBUG:boto:Path: /xryqqk-16/?versions&max-keys=999 2024-04-09T22:04:22.296 DEBUG:boto:Data: 2024-04-09T22:04:22.296 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.296 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.296 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.296 DEBUG:boto:Params: {} 2024-04-09T22:04:22.296 DEBUG:boto:Token: None 2024-04-09T22:04:22.296 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/?versions 2024-04-09T22:04:22.296 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM= 2024-04-09T22:04:22.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I3/WmSd0Uh22DyUlEgSMUOEAqQM=', 'Content-Length': '0'} 2024-04-09T22:04:22.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e48afa472e426a15-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.299 DEBUG:boto:b'xryqqk-16999falseobjnulltrue2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjWrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SFfalse2024-04-09T22:03:06.202Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:04:22.300 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.300 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.300 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.300 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.300 DEBUG:boto:Method: GET 2024-04-09T22:04:22.300 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.300 DEBUG:boto:Data: 2024-04-09T22:04:22.300 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.300 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.300 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.301 DEBUG:boto:Params: {} 2024-04-09T22:04:22.301 DEBUG:boto:Token: None 2024-04-09T22:04:22.301 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.302 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc= 2024-04-09T22:04:22.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc=', 'Content-Length': '0'} 2024-04-09T22:04:22.304 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ab078e0e4e06dd7-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.304 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.304 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.304 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.304 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.304 DEBUG:boto:Method: GET 2024-04-09T22:04:22.304 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.304 DEBUG:boto:Data: 2024-04-09T22:04:22.304 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.305 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.305 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.305 DEBUG:boto:Params: {} 2024-04-09T22:04:22.305 DEBUG:boto:Token: None 2024-04-09T22:04:22.305 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.305 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc= 2024-04-09T22:04:22.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc=', 'Content-Length': '0'} 2024-04-09T22:04:22.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d23570ce1c2f6da7-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.307 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.307 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.308 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.308 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.308 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.308 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.308 DEBUG:boto:Data: 2024-04-09T22:04:22.308 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.308 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.308 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.308 DEBUG:boto:Params: {} 2024-04-09T22:04:22.308 DEBUG:boto:Token: None 2024-04-09T22:04:22.308 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.309 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HVF8DO15ij9OrVRXfAldwCBAgJ8= 2024-04-09T22:04:22.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HVF8DO15ij9OrVRXfAldwCBAgJ8=', 'Content-Length': '0'} 2024-04-09T22:04:22.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003669db2e65c2e20d-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.310 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.311 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.311 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.311 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.311 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.311 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.311 DEBUG:boto:Data: 2024-04-09T22:04:22.311 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.311 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.311 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.311 DEBUG:boto:Params: {} 2024-04-09T22:04:22.312 DEBUG:boto:Token: None 2024-04-09T22:04:22.312 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.312 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HVF8DO15ij9OrVRXfAldwCBAgJ8= 2024-04-09T22:04:22.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HVF8DO15ij9OrVRXfAldwCBAgJ8=', 'Content-Length': '0'} 2024-04-09T22:04:22.313 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008ee5515ba6edb510-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:04:22.314 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.314 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.314 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.314 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.314 DEBUG:boto:Method: GET 2024-04-09T22:04:22.314 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.314 DEBUG:boto:Data: 2024-04-09T22:04:22.314 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.315 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.315 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.315 DEBUG:boto:Params: {} 2024-04-09T22:04:22.315 DEBUG:boto:Token: None 2024-04-09T22:04:22.315 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.315 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc= 2024-04-09T22:04:22.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc=', 'Content-Length': '0'} 2024-04-09T22:04:22.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003516184226cd82d3-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.317 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.317 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.317 DEBUG:boto:path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.317 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.317 DEBUG:boto:Method: GET 2024-04-09T22:04:22.318 DEBUG:boto:Path: /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.318 DEBUG:boto:Data: 2024-04-09T22:04:22.318 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.318 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.318 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.318 DEBUG:boto:Params: {} 2024-04-09T22:04:22.318 DEBUG:boto:Token: None 2024-04-09T22:04:22.318 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=null 2024-04-09T22:04:22.319 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc= 2024-04-09T22:04:22.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Je5z5IN12uvrTHt66MI1DKhX9oc=', 'Content-Length': '0'} 2024-04-09T22:04:22.320 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c2926595a85035c8-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.321 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.321 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.321 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.321 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.321 DEBUG:boto:Method: GET 2024-04-09T22:04:22.321 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.321 DEBUG:boto:Data: 2024-04-09T22:04:22.321 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.321 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.321 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.322 DEBUG:boto:Params: {} 2024-04-09T22:04:22.322 DEBUG:boto:Token: None 2024-04-09T22:04:22.322 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.322 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f2a05e26329f2c5-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.325 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.325 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.325 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.326 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.326 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.326 DEBUG:boto:Method: GET 2024-04-09T22:04:22.326 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.326 DEBUG:boto:Data: 2024-04-09T22:04:22.326 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.326 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.326 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.326 DEBUG:boto:Params: {} 2024-04-09T22:04:22.326 DEBUG:boto:Token: None 2024-04-09T22:04:22.327 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.327 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000011167c4372358f-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.370 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.371 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.371 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.371 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.371 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.371 DEBUG:boto:Data: 2024-04-09T22:04:22.371 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.371 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.371 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.371 DEBUG:boto:Params: {} 2024-04-09T22:04:22.372 DEBUG:boto:Token: None 2024-04-09T22:04:22.372 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.372 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE= 2024-04-09T22:04:22.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE=', 'Content-Length': '0'} 2024-04-09T22:04:22.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb1e2412e32fe9cc-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.374 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.374 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.374 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.374 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.374 DEBUG:boto:Data: 2024-04-09T22:04:22.375 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.375 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.375 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.375 DEBUG:boto:Params: {} 2024-04-09T22:04:22.375 DEBUG:boto:Token: None 2024-04-09T22:04:22.375 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.375 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE= 2024-04-09T22:04:22.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE=', 'Content-Length': '0'} 2024-04-09T22:04:22.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005836ba91814bc3a3-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:04:22.377 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.377 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.378 DEBUG:boto:Method: GET 2024-04-09T22:04:22.378 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.378 DEBUG:boto:Data: 2024-04-09T22:04:22.378 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.378 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.378 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.378 DEBUG:boto:Params: {} 2024-04-09T22:04:22.378 DEBUG:boto:Token: None 2024-04-09T22:04:22.378 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.379 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA= 2024-04-09T22:04:22.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA=', 'Content-Length': '0'} 2024-04-09T22:04:22.380 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fce6baddac5573b8-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.381 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.381 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.381 DEBUG:boto:Method: GET 2024-04-09T22:04:22.381 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.381 DEBUG:boto:Data: 2024-04-09T22:04:22.381 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.381 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.381 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.381 DEBUG:boto:Params: {} 2024-04-09T22:04:22.381 DEBUG:boto:Token: None 2024-04-09T22:04:22.381 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.382 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA= 2024-04-09T22:04:22.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA=', 'Content-Length': '0'} 2024-04-09T22:04:22.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008997b3ba66d4bc16-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.384 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.384 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.384 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.384 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.384 DEBUG:boto:Method: GET 2024-04-09T22:04:22.384 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.384 DEBUG:boto:Data: 2024-04-09T22:04:22.384 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.384 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.384 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.384 DEBUG:boto:Params: {} 2024-04-09T22:04:22.385 DEBUG:boto:Token: None 2024-04-09T22:04:22.385 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.385 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000363b1ad7ebd3055c-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.387 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.388 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.388 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.388 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.388 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.388 DEBUG:boto:Method: GET 2024-04-09T22:04:22.388 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.388 DEBUG:boto:Data: 2024-04-09T22:04:22.388 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.388 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.388 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.388 DEBUG:boto:Params: {} 2024-04-09T22:04:22.389 DEBUG:boto:Token: None 2024-04-09T22:04:22.389 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.389 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad57a653f575cf46-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.391 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:04:22.392 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.392 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.392 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.392 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.392 DEBUG:boto:Method: GET 2024-04-09T22:04:22.392 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.392 DEBUG:boto:Data: 2024-04-09T22:04:22.392 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.392 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.392 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.392 DEBUG:boto:Params: {} 2024-04-09T22:04:22.393 DEBUG:boto:Token: None 2024-04-09T22:04:22.393 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.393 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I= 2024-04-09T22:04:22.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I=', 'Content-Length': '0'} 2024-04-09T22:04:22.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e688ae2b14a61808-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.395 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.395 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.395 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.395 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.395 DEBUG:boto:Method: GET 2024-04-09T22:04:22.395 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.395 DEBUG:boto:Data: 2024-04-09T22:04:22.395 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.395 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.395 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.395 DEBUG:boto:Params: {} 2024-04-09T22:04:22.396 DEBUG:boto:Token: None 2024-04-09T22:04:22.396 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.396 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I= 2024-04-09T22:04:22.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I=', 'Content-Length': '0'} 2024-04-09T22:04:22.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000449e7f4b70f5bb4e-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.398 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.398 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.398 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.398 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.398 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.398 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.398 DEBUG:boto:Data: 2024-04-09T22:04:22.398 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.398 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.398 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.399 DEBUG:boto:Params: {} 2024-04-09T22:04:22.399 DEBUG:boto:Token: None 2024-04-09T22:04:22.399 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.399 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Wc6NLdqieKiPXnuF1+GuNdu0Rz0= 2024-04-09T22:04:22.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Wc6NLdqieKiPXnuF1+GuNdu0Rz0=', 'Content-Length': '0'} 2024-04-09T22:04:22.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000349e63044275ee5-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.401 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.401 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.401 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.401 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.401 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.401 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.401 DEBUG:boto:Data: 2024-04-09T22:04:22.401 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.401 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.401 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.401 DEBUG:boto:Params: {} 2024-04-09T22:04:22.402 DEBUG:boto:Token: None 2024-04-09T22:04:22.402 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.402 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Wc6NLdqieKiPXnuF1+GuNdu0Rz0= 2024-04-09T22:04:22.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Wc6NLdqieKiPXnuF1+GuNdu0Rz0=', 'Content-Length': '0'} 2024-04-09T22:04:22.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003d3b0d256accbf71-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:04:22.404 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.404 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.404 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.404 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.404 DEBUG:boto:Method: GET 2024-04-09T22:04:22.404 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.404 DEBUG:boto:Data: 2024-04-09T22:04:22.405 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.405 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.405 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.405 DEBUG:boto:Params: {} 2024-04-09T22:04:22.405 DEBUG:boto:Token: None 2024-04-09T22:04:22.405 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.405 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I= 2024-04-09T22:04:22.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I=', 'Content-Length': '0'} 2024-04-09T22:04:22.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005c037a4a6096de03-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.407 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.407 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.407 DEBUG:boto:path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.407 DEBUG:boto:auth_path=/xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.408 DEBUG:boto:Method: GET 2024-04-09T22:04:22.408 DEBUG:boto:Path: /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.408 DEBUG:boto:Data: 2024-04-09T22:04:22.408 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.408 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.408 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.408 DEBUG:boto:Params: {} 2024-04-09T22:04:22.408 DEBUG:boto:Token: None 2024-04-09T22:04:22.408 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj?versionId=WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF 2024-04-09T22:04:22.408 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I= 2024-04-09T22:04:22.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VXlH84P/J8P+HxSeTVWvuZmqD+I=', 'Content-Length': '0'} 2024-04-09T22:04:22.410 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:06 GMT'), ('x-amz-version-id', 'WrnqU-hhUpZ26Dfvd7Jb0fk6Th2k-SF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f33e636fbceeda04-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.410 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.410 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.410 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.411 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.411 DEBUG:boto:Method: GET 2024-04-09T22:04:22.411 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.411 DEBUG:boto:Data: 2024-04-09T22:04:22.411 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.411 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.411 DEBUG:boto:Params: {} 2024-04-09T22:04:22.411 DEBUG:boto:Token: None 2024-04-09T22:04:22.411 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.411 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4a6a3b31df344f1-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.458 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.459 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.459 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.459 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.459 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.459 DEBUG:boto:Method: GET 2024-04-09T22:04:22.459 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.459 DEBUG:boto:Data: 2024-04-09T22:04:22.460 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.460 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.460 DEBUG:boto:Params: {} 2024-04-09T22:04:22.460 DEBUG:boto:Token: None 2024-04-09T22:04:22.460 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050cdd361aeab5829-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.463 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.463 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.463 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.464 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.464 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.464 DEBUG:boto:Data: 2024-04-09T22:04:22.464 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.464 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.464 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.464 DEBUG:boto:Params: {} 2024-04-09T22:04:22.464 DEBUG:boto:Token: None 2024-04-09T22:04:22.464 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE= 2024-04-09T22:04:22.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE=', 'Content-Length': '0'} 2024-04-09T22:04:22.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d2e77851699f7cd-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.466 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.467 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.467 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.467 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.467 DEBUG:boto:Data: 2024-04-09T22:04:22.467 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.467 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.467 DEBUG:boto:Params: {} 2024-04-09T22:04:22.467 DEBUG:boto:Token: None 2024-04-09T22:04:22.467 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE= 2024-04-09T22:04:22.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KyBqPvpqabVLzv7+4zrqzuY66KE=', 'Content-Length': '0'} 2024-04-09T22:04:22.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000856bd144bd42dfb-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:04:22.470 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.470 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.470 DEBUG:boto:Method: GET 2024-04-09T22:04:22.470 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.470 DEBUG:boto:Data: 2024-04-09T22:04:22.470 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.470 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.470 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.470 DEBUG:boto:Params: {} 2024-04-09T22:04:22.470 DEBUG:boto:Token: None 2024-04-09T22:04:22.471 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA= 2024-04-09T22:04:22.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA=', 'Content-Length': '0'} 2024-04-09T22:04:22.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-16:a5723776-f142-428c-a6df-d7bf949257da.4372.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000047458cb6311d1e4f-006615bb66-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.472 DEBUG:boto:path=/xryqqk-16/obj 2024-04-09T22:04:22.473 DEBUG:boto:auth_path=/xryqqk-16/obj 2024-04-09T22:04:22.473 DEBUG:boto:Method: GET 2024-04-09T22:04:22.473 DEBUG:boto:Path: /xryqqk-16/obj 2024-04-09T22:04:22.473 DEBUG:boto:Data: 2024-04-09T22:04:22.473 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.473 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.473 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.473 DEBUG:boto:Params: {} 2024-04-09T22:04:22.473 DEBUG:boto:Token: None 2024-04-09T22:04:22.473 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/obj 2024-04-09T22:04:22.474 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA= 2024-04-09T22:04:22.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xNJ8G2IlZtmYOtVPDISfUnlZHA=', 'Content-Length': '0'} 2024-04-09T22:04:22.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ffb9342d31403618-006615bb66-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.475 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.475 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.476 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.476 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.476 DEBUG:boto:Method: GET 2024-04-09T22:04:22.476 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.476 DEBUG:boto:Data: 2024-04-09T22:04:22.476 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.476 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.476 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.476 DEBUG:boto:Params: {} 2024-04-09T22:04:22.476 DEBUG:boto:Token: None 2024-04-09T22:04:22.477 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.477 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e92eb3e4cd91aede-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.479 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.479 DEBUG:boto:path=/xryqqk-16/ 2024-04-09T22:04:22.479 DEBUG:boto:auth_path=/xryqqk-16/ 2024-04-09T22:04:22.480 DEBUG:boto:path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.480 DEBUG:boto:auth_path=/xryqqk-16/?prefix=obj 2024-04-09T22:04:22.480 DEBUG:boto:Method: GET 2024-04-09T22:04:22.480 DEBUG:boto:Path: /xryqqk-16/?prefix=obj 2024-04-09T22:04:22.480 DEBUG:boto:Data: 2024-04-09T22:04:22.480 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.480 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.480 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.480 DEBUG:boto:Params: {} 2024-04-09T22:04:22.480 DEBUG:boto:Token: None 2024-04-09T22:04:22.480 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-16/ 2024-04-09T22:04:22.481 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc= 2024-04-09T22:04:22.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DumKnzr1MWQUWpaDKjGkTmM/+jc=', 'Content-Length': '0'} 2024-04-09T22:04:22.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c33a7caa5cb65fe1-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.526 DEBUG:boto:b'xryqqk-16obj1000falseobj2024-04-09T22:03:41.974Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:04:22.527 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-16 zones={a2, a1} 2024-04-09T22:04:22.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-04-09T22:04:22.528 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-17 2024-04-09T22:04:22.529 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:04:22.529 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:04:22.529 DEBUG:boto:Method: PUT 2024-04-09T22:04:22.529 DEBUG:boto:Path: /xryqqk-17/ 2024-04-09T22:04:22.529 DEBUG:boto:Data: 2024-04-09T22:04:22.529 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.529 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.529 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.529 DEBUG:boto:Params: {} 2024-04-09T22:04:22.530 DEBUG:boto:Token: None 2024-04-09T22:04:22.530 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-17/ 2024-04-09T22:04:22.530 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pBMwuGOIsYq7hvJNmCRuytE/NyM= 2024-04-09T22:04:22.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pBMwuGOIsYq7hvJNmCRuytE/NyM=', 'Content-Length': '0'} 2024-04-09T22:04:22.545 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002424ccabc75f9471-006615bb66-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.545 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-18 2024-04-09T22:04:22.545 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:04:22.545 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:04:22.545 DEBUG:boto:Method: PUT 2024-04-09T22:04:22.545 DEBUG:boto:Path: /xryqqk-18/ 2024-04-09T22:04:22.545 DEBUG:boto:Data: 2024-04-09T22:04:22.545 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.546 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.546 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.546 DEBUG:boto:Params: {} 2024-04-09T22:04:22.546 DEBUG:boto:Token: None 2024-04-09T22:04:22.546 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-18/ 2024-04-09T22:04:22.546 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JAnljRTBNMCvU2zHyBDMEImlOxk= 2024-04-09T22:04:22.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JAnljRTBNMCvU2zHyBDMEImlOxk=', 'Content-Length': '0'} 2024-04-09T22:04:22.570 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000497425e5aaf8b6c7-006615bb66-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.570 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:04:22.571 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:04:22.571 DEBUG:boto:path=/xryqqk-17/?versioning 2024-04-09T22:04:22.571 DEBUG:boto:auth_path=/xryqqk-17/?versioning 2024-04-09T22:04:22.571 DEBUG:boto:Method: PUT 2024-04-09T22:04:22.571 DEBUG:boto:Path: /xryqqk-17/?versioning 2024-04-09T22:04:22.571 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:04:22.571 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.572 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.572 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.572 DEBUG:boto:Params: {} 2024-04-09T22:04:22.572 DEBUG:boto:Token: None 2024-04-09T22:04:22.572 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-17/?versioning 2024-04-09T22:04:22.572 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ze+N5sI9Q/xH0jaSBI6jP8WbL0E= 2024-04-09T22:04:22.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ze+N5sI9Q/xH0jaSBI6jP8WbL0E=', 'Content-Length': '228'} 2024-04-09T22:04:22.578 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004bd849d59a4eec9b-006615bb66-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.578 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:04:22.578 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:04:22.578 DEBUG:boto:path=/xryqqk-18/?versioning 2024-04-09T22:04:22.578 DEBUG:boto:auth_path=/xryqqk-18/?versioning 2024-04-09T22:04:22.578 DEBUG:boto:Method: PUT 2024-04-09T22:04:22.579 DEBUG:boto:Path: /xryqqk-18/?versioning 2024-04-09T22:04:22.579 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:04:22.579 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.579 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.579 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.579 DEBUG:boto:Params: {} 2024-04-09T22:04:22.579 DEBUG:boto:Token: None 2024-04-09T22:04:22.579 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-18/?versioning 2024-04-09T22:04:22.580 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5JqSUbMOYqZi8SMpFysdkikcU54= 2024-04-09T22:04:22.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5JqSUbMOYqZi8SMpFysdkikcU54=', 'Content-Length': '228'} 2024-04-09T22:04:22.588 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c92ef8ee30f91da5-006615bb66-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:04:22.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:04:22.588 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:04:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:04:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:04:22.678 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.692 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700262.585047_291.1", 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:04:22.585047Z" 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:04:22.693 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:04:22.695 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700262.585047_291.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:04:22.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:04:22.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:04:22.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:04:22.695 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:04:22.748 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:04:22.749 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:04:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:22.775+0000 7ff125758780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:04:22.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:22.776+0000 7ff125758780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:04:22.789 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:04:22.789 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:04:22.789 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:04:22.790 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.791 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:04:22.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700262.585047_291.1", 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:04:22.585047Z", 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:04:22.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:04:22.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:04:22Z" 2024-04-09T22:04:22.795 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:04:22.796 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700262.585047_291.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:04:22.585047Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:04:22Z" } 2024-04-09T22:04:22.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700262.585047_291.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:04:22.585047Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:04:22.796 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700262.585047_291.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:04:22.797 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700262.585047_291.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:04:22.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:04:22.797 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:04:22.797 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:04:22.797 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.797 DEBUG:boto:Path: /xryqqk-17/ 2024-04-09T22:04:22.797 DEBUG:boto:Data: 2024-04-09T22:04:22.797 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.797 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.797 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.797 DEBUG:boto:Params: {} 2024-04-09T22:04:22.798 DEBUG:boto:Token: None 2024-04-09T22:04:22.798 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-17/ 2024-04-09T22:04:22.798 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5eqG8aVSrJesU23g6yb1xTL6sF4= 2024-04-09T22:04:22.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5eqG8aVSrJesU23g6yb1xTL6sF4=', 'Content-Length': '0'} 2024-04-09T22:04:22.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d004bcd88e27e7c-006615bb66-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.800 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:04:22.801 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:04:22.801 DEBUG:boto:Method: PUT 2024-04-09T22:04:22.801 DEBUG:boto:Path: /xryqqk-17/obj 2024-04-09T22:04:22.801 DEBUG:boto:Data: 2024-04-09T22:04:22.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:04:22.801 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.801 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.801 DEBUG:boto:Params: {} 2024-04-09T22:04:22.801 DEBUG:boto:Token: None 2024-04-09T22:04:22.801 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-17/obj 2024-04-09T22:04:22.802 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4zA2ogvu9hVqzvlnU0zmDr3brO8= 2024-04-09T22:04:22.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4zA2ogvu9hVqzvlnU0zmDr3brO8='} 2024-04-09T22:04:22.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-amz-request-id', 'tx0000006585dc98d8980e7-006615bb66-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.812 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:04:22.812 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:04:22.812 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.812 DEBUG:boto:Path: /xryqqk-17/ 2024-04-09T22:04:22.813 DEBUG:boto:Data: 2024-04-09T22:04:22.813 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.813 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.813 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.813 DEBUG:boto:Params: {} 2024-04-09T22:04:22.813 DEBUG:boto:Token: None 2024-04-09T22:04:22.813 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-17/ 2024-04-09T22:04:22.813 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5eqG8aVSrJesU23g6yb1xTL6sF4= 2024-04-09T22:04:22.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5eqG8aVSrJesU23g6yb1xTL6sF4=', 'Content-Length': '0'} 2024-04-09T22:04:22.815 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c18f1ebcb01caf04-006615bb66-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.816 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:04:22.816 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:04:22.816 DEBUG:boto:Method: DELETE 2024-04-09T22:04:22.816 DEBUG:boto:Path: /xryqqk-17/obj 2024-04-09T22:04:22.816 DEBUG:boto:Data: 2024-04-09T22:04:22.816 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.816 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:04:22.816 DEBUG:boto:Port: 8000 2024-04-09T22:04:22.816 DEBUG:boto:Params: {} 2024-04-09T22:04:22.817 DEBUG:boto:Token: None 2024-04-09T22:04:22.817 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-17/obj 2024-04-09T22:04:22.817 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JgHdalFyRsiB+J48x8tSuWVIh8k= 2024-04-09T22:04:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JgHdalFyRsiB+J48x8tSuWVIh8k=', 'Content-Length': '0'} 2024-04-09T22:04:22.823 DEBUG:boto:Response headers: [('x-amz-version-id', 'oD.wqYok7lobWRU2OiUpSTbCJ2iX4XX'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000006bd9f072b30bcf77-006615bb66-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.823 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:04:22.823 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:04:22.823 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.823 DEBUG:boto:Path: /xryqqk-18/ 2024-04-09T22:04:22.823 DEBUG:boto:Data: 2024-04-09T22:04:22.823 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.824 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.824 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.824 DEBUG:boto:Params: {} 2024-04-09T22:04:22.824 DEBUG:boto:Token: None 2024-04-09T22:04:22.824 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-18/ 2024-04-09T22:04:22.824 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IcZY0njOq98X9vkiSMt+BxSb8h4= 2024-04-09T22:04:22.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IcZY0njOq98X9vkiSMt+BxSb8h4=', 'Content-Length': '0'} 2024-04-09T22:04:22.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ac15cd04e63f346-006615bb66-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.826 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:04:22.827 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:04:22.827 DEBUG:boto:Method: PUT 2024-04-09T22:04:22.827 DEBUG:boto:Path: /xryqqk-18/obj 2024-04-09T22:04:22.827 DEBUG:boto:Data: 2024-04-09T22:04:22.827 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:04:22.827 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.827 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.827 DEBUG:boto:Params: {} 2024-04-09T22:04:22.827 DEBUG:boto:Token: None 2024-04-09T22:04:22.827 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-18/obj 2024-04-09T22:04:22.828 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:39dMld6V9/PBI+dpL1ytXzpNJSs= 2024-04-09T22:04:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:39dMld6V9/PBI+dpL1ytXzpNJSs='} 2024-04-09T22:04:22.837 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-amz-request-id', 'tx0000006062f0067d6ddd3-006615bb66-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.837 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:04:22.837 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:04:22.837 DEBUG:boto:Method: HEAD 2024-04-09T22:04:22.838 DEBUG:boto:Path: /xryqqk-18/ 2024-04-09T22:04:22.838 DEBUG:boto:Data: 2024-04-09T22:04:22.838 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.838 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.838 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.838 DEBUG:boto:Params: {} 2024-04-09T22:04:22.838 DEBUG:boto:Token: None 2024-04-09T22:04:22.838 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-18/ 2024-04-09T22:04:22.838 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IcZY0njOq98X9vkiSMt+BxSb8h4= 2024-04-09T22:04:22.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IcZY0njOq98X9vkiSMt+BxSb8h4=', 'Content-Length': '0'} 2024-04-09T22:04:22.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f84b1141af8b93e6-006615bb66-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.841 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:04:22.841 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:04:22.841 DEBUG:boto:Method: DELETE 2024-04-09T22:04:22.841 DEBUG:boto:Path: /xryqqk-18/obj 2024-04-09T22:04:22.841 DEBUG:boto:Data: 2024-04-09T22:04:22.841 DEBUG:boto:Headers: {} 2024-04-09T22:04:22.841 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:04:22.841 DEBUG:boto:Port: 8001 2024-04-09T22:04:22.841 DEBUG:boto:Params: {} 2024-04-09T22:04:22.842 DEBUG:boto:Token: None 2024-04-09T22:04:22.842 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 22:04:22 GMT /xryqqk-18/obj 2024-04-09T22:04:22.842 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OVilJ6HTzRfciVPRXHqDjupxtgc= 2024-04-09T22:04:22.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:04:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OVilJ6HTzRfciVPRXHqDjupxtgc=', 'Content-Length': '0'} 2024-04-09T22:04:22.848 DEBUG:boto:Response headers: [('x-amz-version-id', '.ixaqKXVtI6FxxJ0DDAU9HJyE5m5J4G'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000ee8d5c174b006c1f-006615bb66-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:04:22.848 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-17 2024-04-09T22:04:22.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:04:22.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:04:22.849 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:04:22.905 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:04:22.905 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:04:22.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:22.938+0000 7f20d549f780 1 waiting to reach incremental sync.. 2024-04-09T22:04:27.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:27.939+0000 7f20d549f780 1 waiting to reach incremental sync.. 2024-04-09T22:04:32.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:32.939+0000 7f20d549f780 1 waiting to reach incremental sync.. 2024-04-09T22:04:37.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:37.940+0000 7f20d549f780 1 waiting to reach incremental sync.. 2024-04-09T22:04:42.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:42.943+0000 7f20d549f780 1 bucket sync caught up with source: 2024-04-09T22:04:42.944 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.26.12, , ] 2024-04-09T22:04:42.944 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.26.12, , ] 2024-04-09T22:04:42.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:42.943+0000 7f20d549f780 0 bucket checkpoint complete 2024-04-09T22:04:42.960 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-17 2024-04-09T22:04:42.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:04:42.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:04:42.960 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:04:43.023 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:04:43.023 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:04:43.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:43.061+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:04:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:48.061+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:04:53.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:53.061+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:04:58.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:04:58.061+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:03.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:03.062+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:08.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:08.063+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:13.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:13.064+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:18.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:18.064+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:23.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:23.065+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:28.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:28.066+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:33.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:33.066+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:38.067+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:43.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:43.067+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:48.068+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:53.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:53.069+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:05:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:05:58.069+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:03.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:03.070+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:08.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:08.070+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:13.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:13.071+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:18.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:18.072+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:23.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:23.073+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:28.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:28.074+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:33.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:33.073+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:38.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:38.074+0000 7f699d1c7780 1 waiting to reach incremental sync.. 2024-04-09T22:06:43.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.075+0000 7f699d1c7780 1 bucket sync caught up with source: 2024-04-09T22:06:43.077 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000002.32.6, , ] 2024-04-09T22:06:43.077 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000002.32.6, , ] 2024-04-09T22:06:43.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.075+0000 7f699d1c7780 0 bucket checkpoint complete 2024-04-09T22:06:43.095 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-17 zones={a2, a1} 2024-04-09T22:06:43.095 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.095 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.095 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.095 DEBUG:boto:Path: /xryqqk-17/ 2024-04-09T22:06:43.095 DEBUG:boto:Data: 2024-04-09T22:06:43.095 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.096 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.096 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.096 DEBUG:boto:Params: {} 2024-04-09T22:06:43.096 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:06:43.096 DEBUG:boto:Token: None 2024-04-09T22:06:43.097 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/ 2024-04-09T22:06:43.097 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FjKfIuBuBgSQV4lIIdnWsN/lLHM= 2024-04-09T22:06:43.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FjKfIuBuBgSQV4lIIdnWsN/lLHM=', 'Content-Length': '0'} 2024-04-09T22:06:43.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a1cc0b0a2eb5722-006615bbf3-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.102 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.102 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.102 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.102 DEBUG:boto:Path: /xryqqk-17/ 2024-04-09T22:06:43.103 DEBUG:boto:Data: 2024-04-09T22:06:43.103 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.103 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.103 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.103 DEBUG:boto:Params: {} 2024-04-09T22:06:43.103 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:06:43.103 DEBUG:boto:Token: None 2024-04-09T22:06:43.103 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/ 2024-04-09T22:06:43.103 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FjKfIuBuBgSQV4lIIdnWsN/lLHM= 2024-04-09T22:06:43.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FjKfIuBuBgSQV4lIIdnWsN/lLHM=', 'Content-Length': '0'} 2024-04-09T22:06:43.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000194aa974844bf4c4-006615bbf3-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.107 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:06:43.107 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.108 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.108 DEBUG:boto:path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.108 DEBUG:boto:auth_path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.108 DEBUG:boto:Method: GET 2024-04-09T22:06:43.108 DEBUG:boto:Path: /xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.108 DEBUG:boto:Data: 2024-04-09T22:06:43.108 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.108 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.108 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.108 DEBUG:boto:Params: {} 2024-04-09T22:06:43.109 DEBUG:boto:Token: None 2024-04-09T22:06:43.109 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/?versions 2024-04-09T22:06:43.109 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk= 2024-04-09T22:06:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk=', 'Content-Length': '0'} 2024-04-09T22:06:43.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b1910a968c64d69-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.112 DEBUG:boto:b'xryqqk-17999falseobjoD.wqYok7lobWRU2OiUpSTbCJ2iX4XXtrue2024-04-09T22:04:22.819Zrgw-multisite-test-userMultisite Test UserNormalobjNBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5false2024-04-09T22:04:22.803Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.113 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:06:43.113 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.113 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.113 DEBUG:boto:path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.113 DEBUG:boto:auth_path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.114 DEBUG:boto:Method: GET 2024-04-09T22:06:43.114 DEBUG:boto:Path: /xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.114 DEBUG:boto:Data: 2024-04-09T22:06:43.114 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.114 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.114 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.114 DEBUG:boto:Params: {} 2024-04-09T22:06:43.114 DEBUG:boto:Token: None 2024-04-09T22:06:43.114 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/?versions 2024-04-09T22:06:43.115 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk= 2024-04-09T22:06:43.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk=', 'Content-Length': '0'} 2024-04-09T22:06:43.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf4ddd9344e7ec44-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.117 DEBUG:boto:b'xryqqk-17999falseobjoD.wqYok7lobWRU2OiUpSTbCJ2iX4XXtrue2024-04-09T22:04:22.819Zrgw-multisite-test-userMultisite Test UserNormalobjNBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5false2024-04-09T22:04:22.803Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.118 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.118 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.118 DEBUG:boto:path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.118 DEBUG:boto:auth_path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.118 DEBUG:boto:Method: GET 2024-04-09T22:06:43.118 DEBUG:boto:Path: /xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.118 DEBUG:boto:Data: 2024-04-09T22:06:43.118 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.118 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.118 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.119 DEBUG:boto:Params: {} 2024-04-09T22:06:43.119 DEBUG:boto:Token: None 2024-04-09T22:06:43.119 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/?versions 2024-04-09T22:06:43.119 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk= 2024-04-09T22:06:43.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk=', 'Content-Length': '0'} 2024-04-09T22:06:43.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000ea1ad48f66c183-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.121 DEBUG:boto:b'xryqqk-17999falseobjoD.wqYok7lobWRU2OiUpSTbCJ2iX4XXtrue2024-04-09T22:04:22.819Zrgw-multisite-test-userMultisite Test UserNormalobjNBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5false2024-04-09T22:04:22.803Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.122 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.122 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.122 DEBUG:boto:path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.122 DEBUG:boto:auth_path=/xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.122 DEBUG:boto:Method: GET 2024-04-09T22:06:43.122 DEBUG:boto:Path: /xryqqk-17/?versions&max-keys=999 2024-04-09T22:06:43.122 DEBUG:boto:Data: 2024-04-09T22:06:43.122 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.122 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.122 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.123 DEBUG:boto:Params: {} 2024-04-09T22:06:43.123 DEBUG:boto:Token: None 2024-04-09T22:06:43.123 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/?versions 2024-04-09T22:06:43.123 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk= 2024-04-09T22:06:43.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7cpPUli+sKjOE2zPkVNUHIPhPpk=', 'Content-Length': '0'} 2024-04-09T22:06:43.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008446ce1cf7f2be9b-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.125 DEBUG:boto:b'xryqqk-17999falseobjoD.wqYok7lobWRU2OiUpSTbCJ2iX4XXtrue2024-04-09T22:04:22.819Zrgw-multisite-test-userMultisite Test UserNormalobjNBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5false2024-04-09T22:04:22.803Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:06:43.126 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.126 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.126 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.126 DEBUG:boto:Path: /xryqqk-17/obj 2024-04-09T22:06:43.126 DEBUG:boto:Data: 2024-04-09T22:06:43.126 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.126 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.126 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.127 DEBUG:boto:Params: {} 2024-04-09T22:06:43.127 DEBUG:boto:Token: None 2024-04-09T22:06:43.127 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj 2024-04-09T22:06:43.127 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo= 2024-04-09T22:06:43.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo=', 'Content-Length': '0'} 2024-04-09T22:06:43.129 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000212a2185a555a667-006615bbf3-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.129 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.129 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.129 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.129 DEBUG:boto:Path: /xryqqk-17/obj 2024-04-09T22:06:43.130 DEBUG:boto:Data: 2024-04-09T22:06:43.130 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.130 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.130 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.130 DEBUG:boto:Params: {} 2024-04-09T22:06:43.130 DEBUG:boto:Token: None 2024-04-09T22:06:43.130 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj 2024-04-09T22:06:43.130 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo= 2024-04-09T22:06:43.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo=', 'Content-Length': '0'} 2024-04-09T22:06:43.132 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002ed26e36373a30ee-006615bbf3-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:06:43.133 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.133 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.133 DEBUG:boto:path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.133 DEBUG:boto:auth_path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.133 DEBUG:boto:Method: GET 2024-04-09T22:06:43.133 DEBUG:boto:Path: /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.133 DEBUG:boto:Data: 2024-04-09T22:06:43.133 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.133 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.133 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.134 DEBUG:boto:Params: {} 2024-04-09T22:06:43.134 DEBUG:boto:Token: None 2024-04-09T22:06:43.134 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.134 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4= 2024-04-09T22:06:43.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4=', 'Content-Length': '0'} 2024-04-09T22:06:43.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-17:a5723776-f142-428c-a6df-d7bf949257da.4372.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000060590494e714fb7e-006615bbf3-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.136 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.136 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.136 DEBUG:boto:path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.136 DEBUG:boto:auth_path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.136 DEBUG:boto:Method: GET 2024-04-09T22:06:43.136 DEBUG:boto:Path: /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.136 DEBUG:boto:Data: 2024-04-09T22:06:43.137 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.137 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.137 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.137 DEBUG:boto:Params: {} 2024-04-09T22:06:43.137 DEBUG:boto:Token: None 2024-04-09T22:06:43.137 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.137 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4= 2024-04-09T22:06:43.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4=', 'Content-Length': '0'} 2024-04-09T22:06:43.139 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000097e7f8579d2787a-006615bbf3-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.139 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.139 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.139 DEBUG:boto:path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.140 DEBUG:boto:auth_path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.140 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.140 DEBUG:boto:Path: /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.140 DEBUG:boto:Data: 2024-04-09T22:06:43.140 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.140 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.140 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.140 DEBUG:boto:Params: {} 2024-04-09T22:06:43.140 DEBUG:boto:Token: None 2024-04-09T22:06:43.140 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.141 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9SNUFNwwjoUH9xzAIyFY+PhqOhw= 2024-04-09T22:06:43.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9SNUFNwwjoUH9xzAIyFY+PhqOhw=', 'Content-Length': '0'} 2024-04-09T22:06:43.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-17:a5723776-f142-428c-a6df-d7bf949257da.4372.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005690eb922e716f3d-006615bbf3-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.142 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.142 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.143 DEBUG:boto:path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.143 DEBUG:boto:auth_path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.143 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.143 DEBUG:boto:Path: /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.143 DEBUG:boto:Data: 2024-04-09T22:06:43.143 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.143 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.143 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.143 DEBUG:boto:Params: {} 2024-04-09T22:06:43.143 DEBUG:boto:Token: None 2024-04-09T22:06:43.143 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9SNUFNwwjoUH9xzAIyFY+PhqOhw= 2024-04-09T22:06:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9SNUFNwwjoUH9xzAIyFY+PhqOhw=', 'Content-Length': '0'} 2024-04-09T22:06:43.145 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b7a418e28e2de287-006615bbf3-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:06:43.145 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.146 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.146 DEBUG:boto:path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.146 DEBUG:boto:auth_path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.146 DEBUG:boto:Method: GET 2024-04-09T22:06:43.146 DEBUG:boto:Path: /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.146 DEBUG:boto:Data: 2024-04-09T22:06:43.146 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.146 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.146 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.146 DEBUG:boto:Params: {} 2024-04-09T22:06:43.146 DEBUG:boto:Token: None 2024-04-09T22:06:43.147 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.147 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4= 2024-04-09T22:06:43.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4=', 'Content-Length': '0'} 2024-04-09T22:06:43.148 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-17:a5723776-f142-428c-a6df-d7bf949257da.4372.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0105f24856e93f5-006615bbf3-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.149 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.149 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.149 DEBUG:boto:path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.149 DEBUG:boto:auth_path=/xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.149 DEBUG:boto:Method: GET 2024-04-09T22:06:43.149 DEBUG:boto:Path: /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.149 DEBUG:boto:Data: 2024-04-09T22:06:43.149 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.149 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.149 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.150 DEBUG:boto:Params: {} 2024-04-09T22:06:43.150 DEBUG:boto:Token: None 2024-04-09T22:06:43.150 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj?versionId=NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5 2024-04-09T22:06:43.150 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4= 2024-04-09T22:06:43.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AxaQbfaZyyn4h52DSdv9sLWWli4=', 'Content-Length': '0'} 2024-04-09T22:06:43.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', 'NBDKFQ4oSz3rsh-CIZd1UvymFUxu1.5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e888ed15c8ea495e-006615bbf3-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.152 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.152 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.152 DEBUG:boto:path=/xryqqk-17/?prefix=obj 2024-04-09T22:06:43.152 DEBUG:boto:auth_path=/xryqqk-17/?prefix=obj 2024-04-09T22:06:43.152 DEBUG:boto:Method: GET 2024-04-09T22:06:43.152 DEBUG:boto:Path: /xryqqk-17/?prefix=obj 2024-04-09T22:06:43.153 DEBUG:boto:Data: 2024-04-09T22:06:43.153 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.153 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.153 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.153 DEBUG:boto:Params: {} 2024-04-09T22:06:43.153 DEBUG:boto:Token: None 2024-04-09T22:06:43.153 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/ 2024-04-09T22:06:43.153 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JRmp1e7dz7D0PxrqQVvULh/O+w8= 2024-04-09T22:06:43.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JRmp1e7dz7D0PxrqQVvULh/O+w8=', 'Content-Length': '0'} 2024-04-09T22:06:43.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061d2cae924c58eef-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.156 DEBUG:boto:b'xryqqk-17obj1000false' 2024-04-09T22:06:43.156 DEBUG:boto:path=/xryqqk-17/ 2024-04-09T22:06:43.156 DEBUG:boto:auth_path=/xryqqk-17/ 2024-04-09T22:06:43.156 DEBUG:boto:path=/xryqqk-17/?prefix=obj 2024-04-09T22:06:43.156 DEBUG:boto:auth_path=/xryqqk-17/?prefix=obj 2024-04-09T22:06:43.156 DEBUG:boto:Method: GET 2024-04-09T22:06:43.157 DEBUG:boto:Path: /xryqqk-17/?prefix=obj 2024-04-09T22:06:43.157 DEBUG:boto:Data: 2024-04-09T22:06:43.157 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.157 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.157 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.157 DEBUG:boto:Params: {} 2024-04-09T22:06:43.157 DEBUG:boto:Token: None 2024-04-09T22:06:43.157 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/ 2024-04-09T22:06:43.157 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JRmp1e7dz7D0PxrqQVvULh/O+w8= 2024-04-09T22:06:43.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JRmp1e7dz7D0PxrqQVvULh/O+w8=', 'Content-Length': '0'} 2024-04-09T22:06:43.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000718594a12bad1639-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.160 DEBUG:boto:b'xryqqk-17obj1000false' 2024-04-09T22:06:43.160 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.160 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.160 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.160 DEBUG:boto:Path: /xryqqk-17/obj 2024-04-09T22:06:43.160 DEBUG:boto:Data: 2024-04-09T22:06:43.161 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.161 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.161 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.161 DEBUG:boto:Params: {} 2024-04-09T22:06:43.161 DEBUG:boto:Token: None 2024-04-09T22:06:43.161 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj 2024-04-09T22:06:43.161 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo= 2024-04-09T22:06:43.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo=', 'Content-Length': '0'} 2024-04-09T22:06:43.163 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000083479c8817cf2983-006615bbf3-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.163 DEBUG:boto:path=/xryqqk-17/obj 2024-04-09T22:06:43.163 DEBUG:boto:auth_path=/xryqqk-17/obj 2024-04-09T22:06:43.163 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.164 DEBUG:boto:Path: /xryqqk-17/obj 2024-04-09T22:06:43.164 DEBUG:boto:Data: 2024-04-09T22:06:43.164 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.164 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.164 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.164 DEBUG:boto:Params: {} 2024-04-09T22:06:43.164 DEBUG:boto:Token: None 2024-04-09T22:06:43.164 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-17/obj 2024-04-09T22:06:43.164 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo= 2024-04-09T22:06:43.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wtcBi0orjd7e0qlB9Ld9CroRWVo=', 'Content-Length': '0'} 2024-04-09T22:06:43.166 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000e2bb2d0f522f399-006615bbf3-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-17 zones={a2, a1} 2024-04-09T22:06:43.166 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-18 2024-04-09T22:06:43.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:06:43.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:06:43.167 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:06:43.228 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:06:43.228 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:06:43.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.264+0000 7fbd48aba780 1 bucket sync caught up with source: 2024-04-09T22:06:43.266 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000002.17.6, , ] 2024-04-09T22:06:43.266 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000002.17.6, , ] 2024-04-09T22:06:43.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.264+0000 7fbd48aba780 0 bucket checkpoint complete 2024-04-09T22:06:43.278 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-18 2024-04-09T22:06:43.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:06:43.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:06:43.279 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:06:43.336 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:06:43.336 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:06:43.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.370+0000 7fb1ef1f4780 1 bucket sync caught up with source: 2024-04-09T22:06:43.371 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.34.12, , ] 2024-04-09T22:06:43.371 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.34.12, , ] 2024-04-09T22:06:43.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.370+0000 7fb1ef1f4780 0 bucket checkpoint complete 2024-04-09T22:06:43.384 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-18 zones={a2, a1} 2024-04-09T22:06:43.384 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.384 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.384 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.384 DEBUG:boto:Path: /xryqqk-18/ 2024-04-09T22:06:43.384 DEBUG:boto:Data: 2024-04-09T22:06:43.384 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.384 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.384 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.384 DEBUG:boto:Params: {} 2024-04-09T22:06:43.385 DEBUG:boto:Token: None 2024-04-09T22:06:43.385 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/ 2024-04-09T22:06:43.385 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GGzlj1X60BYbX2auLTiURIsLpOw= 2024-04-09T22:06:43.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GGzlj1X60BYbX2auLTiURIsLpOw=', 'Content-Length': '0'} 2024-04-09T22:06:43.387 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da61adbb1c269b31-006615bbf3-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.387 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.387 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.387 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.388 DEBUG:boto:Path: /xryqqk-18/ 2024-04-09T22:06:43.388 DEBUG:boto:Data: 2024-04-09T22:06:43.388 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.388 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.388 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.388 DEBUG:boto:Params: {} 2024-04-09T22:06:43.388 DEBUG:boto:Token: None 2024-04-09T22:06:43.388 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/ 2024-04-09T22:06:43.388 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GGzlj1X60BYbX2auLTiURIsLpOw= 2024-04-09T22:06:43.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GGzlj1X60BYbX2auLTiURIsLpOw=', 'Content-Length': '0'} 2024-04-09T22:06:43.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa2b63c4049bf566-006615bbf3-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:06:43.391 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.391 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.391 DEBUG:boto:path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.391 DEBUG:boto:auth_path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.391 DEBUG:boto:Method: GET 2024-04-09T22:06:43.391 DEBUG:boto:Path: /xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.391 DEBUG:boto:Data: 2024-04-09T22:06:43.391 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.391 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.391 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.391 DEBUG:boto:Params: {} 2024-04-09T22:06:43.392 DEBUG:boto:Token: None 2024-04-09T22:06:43.392 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/?versions 2024-04-09T22:06:43.392 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM= 2024-04-09T22:06:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM=', 'Content-Length': '0'} 2024-04-09T22:06:43.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2fbf6e56c3037be-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.442 DEBUG:boto:b'xryqqk-18999falseobj.ixaqKXVtI6FxxJ0DDAU9HJyE5m5J4Gtrue2024-04-09T22:04:22.844Zrgw-multisite-test-userMultisite Test UserNormalobj-kc1FCzs6cWwiBGteNLgX6b7WlfebMDfalse2024-04-09T22:04:22.829Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.443 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:06:43.443 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.443 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.443 DEBUG:boto:path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.443 DEBUG:boto:auth_path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.443 DEBUG:boto:Method: GET 2024-04-09T22:06:43.444 DEBUG:boto:Path: /xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.444 DEBUG:boto:Data: 2024-04-09T22:06:43.444 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.444 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.444 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.444 DEBUG:boto:Params: {} 2024-04-09T22:06:43.444 DEBUG:boto:Token: None 2024-04-09T22:06:43.444 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/?versions 2024-04-09T22:06:43.444 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM= 2024-04-09T22:06:43.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM=', 'Content-Length': '0'} 2024-04-09T22:06:43.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000303fa41d83613dd0-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.447 DEBUG:boto:b'xryqqk-18999falseobj.ixaqKXVtI6FxxJ0DDAU9HJyE5m5J4Gtrue2024-04-09T22:04:22.844Zrgw-multisite-test-userMultisite Test UserNormalobj-kc1FCzs6cWwiBGteNLgX6b7WlfebMDfalse2024-04-09T22:04:22.829Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:06:43.448 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.448 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.448 DEBUG:boto:path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.448 DEBUG:boto:auth_path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.448 DEBUG:boto:Method: GET 2024-04-09T22:06:43.449 DEBUG:boto:Path: /xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.449 DEBUG:boto:Data: 2024-04-09T22:06:43.449 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.449 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.449 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.449 DEBUG:boto:Params: {} 2024-04-09T22:06:43.449 DEBUG:boto:Token: None 2024-04-09T22:06:43.449 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/?versions 2024-04-09T22:06:43.450 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM= 2024-04-09T22:06:43.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM=', 'Content-Length': '0'} 2024-04-09T22:06:43.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062d6445336e7e69d-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.452 DEBUG:boto:b'xryqqk-18999falseobj.ixaqKXVtI6FxxJ0DDAU9HJyE5m5J4Gtrue2024-04-09T22:04:22.844Zrgw-multisite-test-userMultisite Test UserNormalobj-kc1FCzs6cWwiBGteNLgX6b7WlfebMDfalse2024-04-09T22:04:22.829Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.452 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.452 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.453 DEBUG:boto:path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.453 DEBUG:boto:auth_path=/xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.453 DEBUG:boto:Method: GET 2024-04-09T22:06:43.453 DEBUG:boto:Path: /xryqqk-18/?versions&max-keys=999 2024-04-09T22:06:43.453 DEBUG:boto:Data: 2024-04-09T22:06:43.453 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.453 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.453 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.453 DEBUG:boto:Params: {} 2024-04-09T22:06:43.453 DEBUG:boto:Token: None 2024-04-09T22:06:43.454 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/?versions 2024-04-09T22:06:43.454 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM= 2024-04-09T22:06:43.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:57qKr+t2fo5z0fUe3gm3km5kSSM=', 'Content-Length': '0'} 2024-04-09T22:06:43.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eecb2e1aff56ebe1-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.456 DEBUG:boto:b'xryqqk-18999falseobj.ixaqKXVtI6FxxJ0DDAU9HJyE5m5J4Gtrue2024-04-09T22:04:22.844Zrgw-multisite-test-userMultisite Test UserNormalobj-kc1FCzs6cWwiBGteNLgX6b7WlfebMDfalse2024-04-09T22:04:22.829Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:06:43.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:06:43.457 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.457 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.457 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.457 DEBUG:boto:Path: /xryqqk-18/obj 2024-04-09T22:06:43.457 DEBUG:boto:Data: 2024-04-09T22:06:43.457 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.457 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.457 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.457 DEBUG:boto:Params: {} 2024-04-09T22:06:43.457 DEBUG:boto:Token: None 2024-04-09T22:06:43.457 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj 2024-04-09T22:06:43.458 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k= 2024-04-09T22:06:43.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k=', 'Content-Length': '0'} 2024-04-09T22:06:43.460 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000066bd9718b72378b-006615bbf3-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.460 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.460 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.460 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.460 DEBUG:boto:Path: /xryqqk-18/obj 2024-04-09T22:06:43.460 DEBUG:boto:Data: 2024-04-09T22:06:43.461 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.461 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.461 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.461 DEBUG:boto:Params: {} 2024-04-09T22:06:43.461 DEBUG:boto:Token: None 2024-04-09T22:06:43.461 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj 2024-04-09T22:06:43.461 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k= 2024-04-09T22:06:43.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k=', 'Content-Length': '0'} 2024-04-09T22:06:43.463 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007a6acd4189e99f3e-006615bbf3-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:06:43.463 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.463 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.464 DEBUG:boto:path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.464 DEBUG:boto:auth_path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.464 DEBUG:boto:Method: GET 2024-04-09T22:06:43.464 DEBUG:boto:Path: /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.464 DEBUG:boto:Data: 2024-04-09T22:06:43.464 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.464 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.464 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.464 DEBUG:boto:Params: {} 2024-04-09T22:06:43.464 DEBUG:boto:Token: None 2024-04-09T22:06:43.465 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.465 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0= 2024-04-09T22:06:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0=', 'Content-Length': '0'} 2024-04-09T22:06:43.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dc5636d5cefece5b-006615bbf3-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.466 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.467 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.467 DEBUG:boto:path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.467 DEBUG:boto:auth_path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.467 DEBUG:boto:Method: GET 2024-04-09T22:06:43.467 DEBUG:boto:Path: /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.467 DEBUG:boto:Data: 2024-04-09T22:06:43.467 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.467 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.467 DEBUG:boto:Params: {} 2024-04-09T22:06:43.467 DEBUG:boto:Token: None 2024-04-09T22:06:43.468 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0= 2024-04-09T22:06:43.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0=', 'Content-Length': '0'} 2024-04-09T22:06:43.469 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-18:a5723776-f142-428c-a6df-d7bf949257da.4372.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000055214d9d2659244c-006615bbf3-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.469 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.470 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.470 DEBUG:boto:path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.470 DEBUG:boto:auth_path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.470 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.470 DEBUG:boto:Path: /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.470 DEBUG:boto:Data: 2024-04-09T22:06:43.470 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.470 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.470 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.470 DEBUG:boto:Params: {} 2024-04-09T22:06:43.471 DEBUG:boto:Token: None 2024-04-09T22:06:43.471 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3PBk4s8WxKi6o024d8stDUG8S+k= 2024-04-09T22:06:43.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3PBk4s8WxKi6o024d8stDUG8S+k=', 'Content-Length': '0'} 2024-04-09T22:06:43.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2686225dc68431d-006615bbf3-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.472 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.473 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.473 DEBUG:boto:path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.473 DEBUG:boto:auth_path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.473 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.473 DEBUG:boto:Path: /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.473 DEBUG:boto:Data: 2024-04-09T22:06:43.473 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.473 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.473 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.473 DEBUG:boto:Params: {} 2024-04-09T22:06:43.474 DEBUG:boto:Token: None 2024-04-09T22:06:43.474 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.474 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3PBk4s8WxKi6o024d8stDUG8S+k= 2024-04-09T22:06:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3PBk4s8WxKi6o024d8stDUG8S+k=', 'Content-Length': '0'} 2024-04-09T22:06:43.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-18:a5723776-f142-428c-a6df-d7bf949257da.4372.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000efd99ea8311b9cb8-006615bbf3-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:06:43.476 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.476 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.476 DEBUG:boto:path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.476 DEBUG:boto:auth_path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.476 DEBUG:boto:Method: GET 2024-04-09T22:06:43.476 DEBUG:boto:Path: /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.476 DEBUG:boto:Data: 2024-04-09T22:06:43.477 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.477 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.477 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.477 DEBUG:boto:Params: {} 2024-04-09T22:06:43.477 DEBUG:boto:Token: None 2024-04-09T22:06:43.477 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.477 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0= 2024-04-09T22:06:43.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0=', 'Content-Length': '0'} 2024-04-09T22:06:43.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000620a29990a3afe08-006615bbf3-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.479 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.479 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.479 DEBUG:boto:path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.479 DEBUG:boto:auth_path=/xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.479 DEBUG:boto:Method: GET 2024-04-09T22:06:43.479 DEBUG:boto:Path: /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.479 DEBUG:boto:Data: 2024-04-09T22:06:43.479 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.480 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.480 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.480 DEBUG:boto:Params: {} 2024-04-09T22:06:43.480 DEBUG:boto:Token: None 2024-04-09T22:06:43.480 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj?versionId=-kc1FCzs6cWwiBGteNLgX6b7WlfebMD 2024-04-09T22:06:43.480 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0= 2024-04-09T22:06:43.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nsEtCdRo7zph99sBgRfdz0NIw/0=', 'Content-Length': '0'} 2024-04-09T22:06:43.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 22:04:22 GMT'), ('x-amz-version-id', '-kc1FCzs6cWwiBGteNLgX6b7WlfebMD'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-18:a5723776-f142-428c-a6df-d7bf949257da.4372.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000277daf3b3b0083db-006615bbf3-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.482 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.482 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.482 DEBUG:boto:path=/xryqqk-18/?prefix=obj 2024-04-09T22:06:43.482 DEBUG:boto:auth_path=/xryqqk-18/?prefix=obj 2024-04-09T22:06:43.483 DEBUG:boto:Method: GET 2024-04-09T22:06:43.483 DEBUG:boto:Path: /xryqqk-18/?prefix=obj 2024-04-09T22:06:43.483 DEBUG:boto:Data: 2024-04-09T22:06:43.483 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.483 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.483 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.483 DEBUG:boto:Params: {} 2024-04-09T22:06:43.483 DEBUG:boto:Token: None 2024-04-09T22:06:43.483 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/ 2024-04-09T22:06:43.483 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lgsafw4yv2zqQJkqNjE5hZp9Jgc= 2024-04-09T22:06:43.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lgsafw4yv2zqQJkqNjE5hZp9Jgc=', 'Content-Length': '0'} 2024-04-09T22:06:43.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017c810cec06f22e1-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.526 DEBUG:boto:b'xryqqk-18obj1000false' 2024-04-09T22:06:43.527 DEBUG:boto:path=/xryqqk-18/ 2024-04-09T22:06:43.527 DEBUG:boto:auth_path=/xryqqk-18/ 2024-04-09T22:06:43.527 DEBUG:boto:path=/xryqqk-18/?prefix=obj 2024-04-09T22:06:43.527 DEBUG:boto:auth_path=/xryqqk-18/?prefix=obj 2024-04-09T22:06:43.527 DEBUG:boto:Method: GET 2024-04-09T22:06:43.527 DEBUG:boto:Path: /xryqqk-18/?prefix=obj 2024-04-09T22:06:43.527 DEBUG:boto:Data: 2024-04-09T22:06:43.527 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.528 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.528 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.528 DEBUG:boto:Params: {} 2024-04-09T22:06:43.528 DEBUG:boto:Token: None 2024-04-09T22:06:43.528 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/ 2024-04-09T22:06:43.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lgsafw4yv2zqQJkqNjE5hZp9Jgc= 2024-04-09T22:06:43.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lgsafw4yv2zqQJkqNjE5hZp9Jgc=', 'Content-Length': '0'} 2024-04-09T22:06:43.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b5a5928a3bf2527-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.531 DEBUG:boto:b'xryqqk-18obj1000false' 2024-04-09T22:06:43.531 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.531 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.531 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.531 DEBUG:boto:Path: /xryqqk-18/obj 2024-04-09T22:06:43.531 DEBUG:boto:Data: 2024-04-09T22:06:43.531 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.531 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.531 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.532 DEBUG:boto:Params: {} 2024-04-09T22:06:43.532 DEBUG:boto:Token: None 2024-04-09T22:06:43.532 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj 2024-04-09T22:06:43.532 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k= 2024-04-09T22:06:43.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k=', 'Content-Length': '0'} 2024-04-09T22:06:43.534 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f429ece7085c7c5b-006615bbf3-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.534 DEBUG:boto:path=/xryqqk-18/obj 2024-04-09T22:06:43.534 DEBUG:boto:auth_path=/xryqqk-18/obj 2024-04-09T22:06:43.534 DEBUG:boto:Method: HEAD 2024-04-09T22:06:43.534 DEBUG:boto:Path: /xryqqk-18/obj 2024-04-09T22:06:43.534 DEBUG:boto:Data: 2024-04-09T22:06:43.534 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.535 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.535 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.535 DEBUG:boto:Params: {} 2024-04-09T22:06:43.535 DEBUG:boto:Token: None 2024-04-09T22:06:43.535 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-18/obj 2024-04-09T22:06:43.535 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k= 2024-04-09T22:06:43.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ro0DImo9/MTPppOuIQIJn8rsC2k=', 'Content-Length': '0'} 2024-04-09T22:06:43.537 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000bc668bf0ef43951a-006615bbf3-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-18 zones={a2, a1} 2024-04-09T22:06:43.537 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-04-09T22:06:43.539 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-19 2024-04-09T22:06:43.539 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:06:43.539 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:06:43.539 DEBUG:boto:Method: PUT 2024-04-09T22:06:43.539 DEBUG:boto:Path: /xryqqk-19/ 2024-04-09T22:06:43.539 DEBUG:boto:Data: 2024-04-09T22:06:43.539 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.540 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.540 DEBUG:boto:Params: {} 2024-04-09T22:06:43.540 DEBUG:boto:Token: None 2024-04-09T22:06:43.540 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-19/ 2024-04-09T22:06:43.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pSEo2n6LVxwJpuuqBuXgCf3bZsY= 2024-04-09T22:06:43.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pSEo2n6LVxwJpuuqBuXgCf3bZsY=', 'Content-Length': '0'} 2024-04-09T22:06:43.556 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f37c9cdca64df920-006615bbf3-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.556 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-20 2024-04-09T22:06:43.556 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:06:43.556 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:06:43.556 DEBUG:boto:Method: PUT 2024-04-09T22:06:43.557 DEBUG:boto:Path: /xryqqk-20/ 2024-04-09T22:06:43.557 DEBUG:boto:Data: 2024-04-09T22:06:43.557 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.557 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.557 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.557 DEBUG:boto:Params: {} 2024-04-09T22:06:43.557 DEBUG:boto:Token: None 2024-04-09T22:06:43.557 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-20/ 2024-04-09T22:06:43.558 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IFljqPzITqAt0GgNMsTxfkDPERM= 2024-04-09T22:06:43.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IFljqPzITqAt0GgNMsTxfkDPERM=', 'Content-Length': '0'} 2024-04-09T22:06:43.579 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ebe2227de7083683-006615bbf3-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.579 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:06:43.579 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:06:43.579 DEBUG:boto:path=/xryqqk-19/?versioning 2024-04-09T22:06:43.579 DEBUG:boto:auth_path=/xryqqk-19/?versioning 2024-04-09T22:06:43.579 DEBUG:boto:Method: PUT 2024-04-09T22:06:43.579 DEBUG:boto:Path: /xryqqk-19/?versioning 2024-04-09T22:06:43.579 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:06:43.580 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.580 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.580 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.580 DEBUG:boto:Params: {} 2024-04-09T22:06:43.580 DEBUG:boto:Token: None 2024-04-09T22:06:43.580 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-19/?versioning 2024-04-09T22:06:43.580 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nbo2stuQ8L7UIWpwTBZv4qAxvuw= 2024-04-09T22:06:43.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbo2stuQ8L7UIWpwTBZv4qAxvuw=', 'Content-Length': '228'} 2024-04-09T22:06:43.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007f4f88b27ff2a5ed-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.585 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:06:43.585 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:06:43.585 DEBUG:boto:path=/xryqqk-19/?versioning 2024-04-09T22:06:43.585 DEBUG:boto:auth_path=/xryqqk-19/?versioning 2024-04-09T22:06:43.586 DEBUG:boto:Method: PUT 2024-04-09T22:06:43.586 DEBUG:boto:Path: /xryqqk-19/?versioning 2024-04-09T22:06:43.586 DEBUG:boto:Data: Suspended Disabled 2024-04-09T22:06:43.586 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.586 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:43.586 DEBUG:boto:Port: 8000 2024-04-09T22:06:43.586 DEBUG:boto:Params: {} 2024-04-09T22:06:43.586 DEBUG:boto:Token: None 2024-04-09T22:06:43.586 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-19/?versioning 2024-04-09T22:06:43.586 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nbo2stuQ8L7UIWpwTBZv4qAxvuw= 2024-04-09T22:06:43.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbo2stuQ8L7UIWpwTBZv4qAxvuw=', 'Content-Length': '230'} 2024-04-09T22:06:43.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009be6f47cbfe10e97-006615bbf3-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.591 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:06:43.591 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:06:43.591 DEBUG:boto:path=/xryqqk-20/?versioning 2024-04-09T22:06:43.591 DEBUG:boto:auth_path=/xryqqk-20/?versioning 2024-04-09T22:06:43.592 DEBUG:boto:Method: PUT 2024-04-09T22:06:43.592 DEBUG:boto:Path: /xryqqk-20/?versioning 2024-04-09T22:06:43.592 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:06:43.592 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.592 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.592 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.592 DEBUG:boto:Params: {} 2024-04-09T22:06:43.592 DEBUG:boto:Token: None 2024-04-09T22:06:43.592 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-20/?versioning 2024-04-09T22:06:43.592 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j/4e2JzyD9TB/DiVaYS3GDbQ2J0= 2024-04-09T22:06:43.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j/4e2JzyD9TB/DiVaYS3GDbQ2J0=', 'Content-Length': '228'} 2024-04-09T22:06:43.599 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000018355452c9913379-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.599 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:06:43.599 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:06:43.600 DEBUG:boto:path=/xryqqk-20/?versioning 2024-04-09T22:06:43.600 DEBUG:boto:auth_path=/xryqqk-20/?versioning 2024-04-09T22:06:43.600 DEBUG:boto:Method: PUT 2024-04-09T22:06:43.600 DEBUG:boto:Path: /xryqqk-20/?versioning 2024-04-09T22:06:43.600 DEBUG:boto:Data: Suspended Disabled 2024-04-09T22:06:43.600 DEBUG:boto:Headers: {} 2024-04-09T22:06:43.600 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:43.600 DEBUG:boto:Port: 8001 2024-04-09T22:06:43.600 DEBUG:boto:Params: {} 2024-04-09T22:06:43.600 DEBUG:boto:Token: None 2024-04-09T22:06:43.601 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:06:43 GMT /xryqqk-20/?versioning 2024-04-09T22:06:43.601 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j/4e2JzyD9TB/DiVaYS3GDbQ2J0= 2024-04-09T22:06:43.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j/4e2JzyD9TB/DiVaYS3GDbQ2J0=', 'Content-Length': '230'} 2024-04-09T22:06:43.607 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005b69047029a23e61-006615bbf3-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:43.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:06:43.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:06:43.608 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:06:43.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:06:43.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:06:43.685 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:06:43.697 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:06:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.697 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700403.605303_327.1", 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:06:43.605303Z" 2024-04-09T22:06:43.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:06:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:06:43.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:43.699 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:06:43.700 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700403.605303_327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:06:43.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:06:43.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:06:43.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:06:43.701 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:06:43.756 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:06:43.757 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:06:43.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.790+0000 7ff6707e9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:06:43.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:43.794+0000 7ff6707e9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:06:43.808 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:06:43.808 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:06:43.808 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:43.809 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:06:43.810 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.811 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700403.547051_312.1", 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:43.812 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:06:43.547051Z", 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:06:43.813 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:43.814 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:06:43Z" 2024-04-09T22:06:43.815 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:06:43.816 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700403.547051_312.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:06:43.547051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:06:43Z" } 2024-04-09T22:06:43.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700403.547051_312.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:06:43.547051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:06:43.816 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700403.605303_327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:06:43.816 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700403.547051_312.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:06:43.816 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700403.605303_327.1 target=1_1712700403.547051_312.1 2024-04-09T22:06:48.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:06:48.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:06:48.818 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:06:48.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:06:48.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:06:48.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:48.906+0000 7fe729c38780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:06:48.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:48.907+0000 7fe729c38780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:06:48.921 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:06:48.921 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:48.922 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:48.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:06:48.924 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700403.605303_327.1", 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:48.925 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:06:43.605303Z", 2024-04-09T22:06:48.926 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:48.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:48.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:48.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:06:48.926 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:06:48.926 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:06:48.927 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:06:48Z" 2024-04-09T22:06:48.928 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:06:48.929 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700403.605303_327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:06:43.605303Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:06:48Z" } 2024-04-09T22:06:48.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700403.605303_327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:06:43.605303Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:06:48.930 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700403.605303_327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:06:48.930 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700403.605303_327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:06:48.930 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:06:48.930 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:06:48.930 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:06:48.930 DEBUG:boto:Method: HEAD 2024-04-09T22:06:48.930 DEBUG:boto:Path: /xryqqk-19/ 2024-04-09T22:06:48.930 DEBUG:boto:Data: 2024-04-09T22:06:48.931 DEBUG:boto:Headers: {} 2024-04-09T22:06:48.931 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:48.931 DEBUG:boto:Port: 8000 2024-04-09T22:06:48.931 DEBUG:boto:Params: {} 2024-04-09T22:06:48.931 DEBUG:boto:Token: None 2024-04-09T22:06:48.931 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-19/ 2024-04-09T22:06:48.931 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jyuSPjdTmsSJS6/MqEkKMDYFTGg= 2024-04-09T22:06:48.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jyuSPjdTmsSJS6/MqEkKMDYFTGg=', 'Content-Length': '0'} 2024-04-09T22:06:48.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083e20e034c85e8af-006615bbf8-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.934 DEBUG:boto:path=/xryqqk-19/obj 2024-04-09T22:06:48.934 DEBUG:boto:auth_path=/xryqqk-19/obj 2024-04-09T22:06:48.934 DEBUG:boto:Method: PUT 2024-04-09T22:06:48.935 DEBUG:boto:Path: /xryqqk-19/obj 2024-04-09T22:06:48.935 DEBUG:boto:Data: 2024-04-09T22:06:48.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:06:48.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:48.935 DEBUG:boto:Port: 8000 2024-04-09T22:06:48.935 DEBUG:boto:Params: {} 2024-04-09T22:06:48.935 DEBUG:boto:Token: None 2024-04-09T22:06:48.935 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-19/obj 2024-04-09T22:06:48.935 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uMGBNoLj495q+a2H1gfD53IaCE8= 2024-04-09T22:06:48.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uMGBNoLj495q+a2H1gfD53IaCE8='} 2024-04-09T22:06:48.941 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d4a688ccb042884-006615bbf8-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.942 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:06:48.942 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:06:48.942 DEBUG:boto:Method: HEAD 2024-04-09T22:06:48.942 DEBUG:boto:Path: /xryqqk-19/ 2024-04-09T22:06:48.942 DEBUG:boto:Data: 2024-04-09T22:06:48.942 DEBUG:boto:Headers: {} 2024-04-09T22:06:48.942 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:48.942 DEBUG:boto:Port: 8000 2024-04-09T22:06:48.942 DEBUG:boto:Params: {} 2024-04-09T22:06:48.943 DEBUG:boto:Token: None 2024-04-09T22:06:48.943 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-19/ 2024-04-09T22:06:48.943 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jyuSPjdTmsSJS6/MqEkKMDYFTGg= 2024-04-09T22:06:48.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jyuSPjdTmsSJS6/MqEkKMDYFTGg=', 'Content-Length': '0'} 2024-04-09T22:06:48.945 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001387e469af2a58ad-006615bbf8-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.945 DEBUG:boto:path=/xryqqk-19/obj 2024-04-09T22:06:48.945 DEBUG:boto:auth_path=/xryqqk-19/obj 2024-04-09T22:06:48.945 DEBUG:boto:Method: DELETE 2024-04-09T22:06:48.945 DEBUG:boto:Path: /xryqqk-19/obj 2024-04-09T22:06:48.946 DEBUG:boto:Data: 2024-04-09T22:06:48.946 DEBUG:boto:Headers: {} 2024-04-09T22:06:48.946 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:06:48.946 DEBUG:boto:Port: 8000 2024-04-09T22:06:48.946 DEBUG:boto:Params: {} 2024-04-09T22:06:48.946 DEBUG:boto:Token: None 2024-04-09T22:06:48.946 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-19/obj 2024-04-09T22:06:48.946 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ddCClwQxSW/qZxkZzUDhALzQ2+A= 2024-04-09T22:06:48.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ddCClwQxSW/qZxkZzUDhALzQ2+A=', 'Content-Length': '0'} 2024-04-09T22:06:48.954 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000003c255985ff9c33b9-006615bbf8-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.955 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:06:48.955 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:06:48.955 DEBUG:boto:Method: HEAD 2024-04-09T22:06:48.955 DEBUG:boto:Path: /xryqqk-20/ 2024-04-09T22:06:48.955 DEBUG:boto:Data: 2024-04-09T22:06:48.955 DEBUG:boto:Headers: {} 2024-04-09T22:06:48.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:48.955 DEBUG:boto:Port: 8001 2024-04-09T22:06:48.955 DEBUG:boto:Params: {} 2024-04-09T22:06:48.956 DEBUG:boto:Token: None 2024-04-09T22:06:48.956 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-20/ 2024-04-09T22:06:48.956 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:o/TBRM9+zvqBwk3w7+ZXsP6uj4Y= 2024-04-09T22:06:48.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:o/TBRM9+zvqBwk3w7+ZXsP6uj4Y=', 'Content-Length': '0'} 2024-04-09T22:06:48.958 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081ba56a6da36903c-006615bbf8-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.959 DEBUG:boto:path=/xryqqk-20/obj 2024-04-09T22:06:48.959 DEBUG:boto:auth_path=/xryqqk-20/obj 2024-04-09T22:06:48.959 DEBUG:boto:Method: PUT 2024-04-09T22:06:48.959 DEBUG:boto:Path: /xryqqk-20/obj 2024-04-09T22:06:48.959 DEBUG:boto:Data: 2024-04-09T22:06:48.959 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-09T22:06:48.959 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:48.960 DEBUG:boto:Port: 8001 2024-04-09T22:06:48.960 DEBUG:boto:Params: {} 2024-04-09T22:06:48.960 DEBUG:boto:Token: None 2024-04-09T22:06:48.960 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-20/obj 2024-04-09T22:06:48.960 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GbdCH0ix16I1Z/1sYdBkoZi4CB0= 2024-04-09T22:06:48.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GbdCH0ix16I1Z/1sYdBkoZi4CB0='} 2024-04-09T22:06:48.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f749aa482d72223f-006615bbf8-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.966 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:06:48.966 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:06:48.966 DEBUG:boto:Method: HEAD 2024-04-09T22:06:48.966 DEBUG:boto:Path: /xryqqk-20/ 2024-04-09T22:06:48.966 DEBUG:boto:Data: 2024-04-09T22:06:48.966 DEBUG:boto:Headers: {} 2024-04-09T22:06:48.966 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:48.966 DEBUG:boto:Port: 8001 2024-04-09T22:06:48.966 DEBUG:boto:Params: {} 2024-04-09T22:06:48.967 DEBUG:boto:Token: None 2024-04-09T22:06:48.967 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-20/ 2024-04-09T22:06:48.967 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:o/TBRM9+zvqBwk3w7+ZXsP6uj4Y= 2024-04-09T22:06:48.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:o/TBRM9+zvqBwk3w7+ZXsP6uj4Y=', 'Content-Length': '0'} 2024-04-09T22:06:48.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004982967e12c34c6b-006615bbf8-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.969 DEBUG:boto:path=/xryqqk-20/obj 2024-04-09T22:06:48.969 DEBUG:boto:auth_path=/xryqqk-20/obj 2024-04-09T22:06:48.970 DEBUG:boto:Method: DELETE 2024-04-09T22:06:48.970 DEBUG:boto:Path: /xryqqk-20/obj 2024-04-09T22:06:48.970 DEBUG:boto:Data: 2024-04-09T22:06:48.970 DEBUG:boto:Headers: {} 2024-04-09T22:06:48.970 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:06:48.970 DEBUG:boto:Port: 8001 2024-04-09T22:06:48.970 DEBUG:boto:Params: {} 2024-04-09T22:06:48.970 DEBUG:boto:Token: None 2024-04-09T22:06:48.970 DEBUG:boto:StringToSign: DELETE Tue, 09 Apr 2024 22:06:48 GMT /xryqqk-20/obj 2024-04-09T22:06:48.971 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Bx7z2+wLti7lp8Gx4NzBkL0svFA= 2024-04-09T22:06:48.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:06:48 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Bx7z2+wLti7lp8Gx4NzBkL0svFA=', 'Content-Length': '0'} 2024-04-09T22:06:48.979 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000043ae37f2b6fe8f9d-006615bbf8-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:06:48.980 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-19 2024-04-09T22:06:48.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:06:48.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:06:48.980 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-19 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:06:49.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:06:49.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:06:49.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:49.072+0000 7fe54eeca780 1 waiting to reach incremental sync.. 2024-04-09T22:06:54.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:54.073+0000 7fe54eeca780 1 waiting to reach incremental sync.. 2024-04-09T22:06:59.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:06:59.073+0000 7fe54eeca780 1 waiting to reach incremental sync.. 2024-04-09T22:07:04.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.075+0000 7fe54eeca780 1 bucket sync caught up with source: 2024-04-09T22:07:04.077 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.32.7, , ] 2024-04-09T22:07:04.077 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.32.7, , ] 2024-04-09T22:07:04.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.075+0000 7fe54eeca780 0 bucket checkpoint complete 2024-04-09T22:07:04.093 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-19 2024-04-09T22:07:04.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:07:04.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:07:04.093 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-19 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:07:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:04.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:04.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.193+0000 7f3d9219b780 1 bucket sync caught up with source: 2024-04-09T22:07:04.195 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.16.14, , ] 2024-04-09T22:07:04.195 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000001.16.14, , ] 2024-04-09T22:07:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.193+0000 7f3d9219b780 0 bucket checkpoint complete 2024-04-09T22:07:04.207 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-19 zones={a2, a1} 2024-04-09T22:07:04.208 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:07:04.208 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:07:04.208 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.208 DEBUG:boto:Path: /xryqqk-19/ 2024-04-09T22:07:04.208 DEBUG:boto:Data: 2024-04-09T22:07:04.208 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.209 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.209 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.209 DEBUG:boto:Params: {} 2024-04-09T22:07:04.209 DEBUG:boto:Token: None 2024-04-09T22:07:04.209 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/ 2024-04-09T22:07:04.209 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r3op9Qmhi+tY8MXwQ7hwf+LLSKY= 2024-04-09T22:07:04.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r3op9Qmhi+tY8MXwQ7hwf+LLSKY=', 'Content-Length': '0'} 2024-04-09T22:07:04.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001519b773c24fbad3-006615bc08-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.212 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:07:04.212 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:07:04.212 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.212 DEBUG:boto:Path: /xryqqk-19/ 2024-04-09T22:07:04.212 DEBUG:boto:Data: 2024-04-09T22:07:04.212 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.212 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.212 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.213 DEBUG:boto:Params: {} 2024-04-09T22:07:04.213 DEBUG:boto:Token: None 2024-04-09T22:07:04.213 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/ 2024-04-09T22:07:04.213 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r3op9Qmhi+tY8MXwQ7hwf+LLSKY= 2024-04-09T22:07:04.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r3op9Qmhi+tY8MXwQ7hwf+LLSKY=', 'Content-Length': '0'} 2024-04-09T22:07:04.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084340494d563a392-006615bc08-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:07:04.215 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:07:04.216 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:07:04.216 DEBUG:boto:path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.216 DEBUG:boto:auth_path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.216 DEBUG:boto:Method: GET 2024-04-09T22:07:04.216 DEBUG:boto:Path: /xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.216 DEBUG:boto:Data: 2024-04-09T22:07:04.216 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.216 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.216 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.216 DEBUG:boto:Params: {} 2024-04-09T22:07:04.216 DEBUG:boto:Token: None 2024-04-09T22:07:04.217 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/?versions 2024-04-09T22:07:04.217 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao= 2024-04-09T22:07:04.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao=', 'Content-Length': '0'} 2024-04-09T22:07:04.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e05fca0ee0198bf9-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.219 DEBUG:boto:b'xryqqk-19999falseobjnulltrue2024-04-09T22:06:48.948Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:07:04.220 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:07:04.220 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:07:04.220 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:07:04.220 DEBUG:boto:path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.220 DEBUG:boto:auth_path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.220 DEBUG:boto:Method: GET 2024-04-09T22:07:04.221 DEBUG:boto:Path: /xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.221 DEBUG:boto:Data: 2024-04-09T22:07:04.221 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.221 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.221 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.221 DEBUG:boto:Params: {} 2024-04-09T22:07:04.221 DEBUG:boto:Token: None 2024-04-09T22:07:04.221 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/?versions 2024-04-09T22:07:04.221 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao= 2024-04-09T22:07:04.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao=', 'Content-Length': '0'} 2024-04-09T22:07:04.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a260270323c4839c-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.223 DEBUG:boto:b'xryqqk-19999falseobjnulltrue2024-04-09T22:06:48.948Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:07:04.224 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:07:04.224 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:07:04.224 DEBUG:boto:path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.224 DEBUG:boto:auth_path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.225 DEBUG:boto:Method: GET 2024-04-09T22:07:04.225 DEBUG:boto:Path: /xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.225 DEBUG:boto:Data: 2024-04-09T22:07:04.225 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.225 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.225 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.225 DEBUG:boto:Params: {} 2024-04-09T22:07:04.225 DEBUG:boto:Token: None 2024-04-09T22:07:04.226 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/?versions 2024-04-09T22:07:04.226 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao= 2024-04-09T22:07:04.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao=', 'Content-Length': '0'} 2024-04-09T22:07:04.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d3759fc850e012b-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.270 DEBUG:boto:b'xryqqk-19999falseobjnulltrue2024-04-09T22:06:48.948Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.271 DEBUG:boto:path=/xryqqk-19/ 2024-04-09T22:07:04.271 DEBUG:boto:auth_path=/xryqqk-19/ 2024-04-09T22:07:04.271 DEBUG:boto:path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.271 DEBUG:boto:auth_path=/xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.271 DEBUG:boto:Method: GET 2024-04-09T22:07:04.271 DEBUG:boto:Path: /xryqqk-19/?versions&max-keys=999 2024-04-09T22:07:04.271 DEBUG:boto:Data: 2024-04-09T22:07:04.272 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.272 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.272 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.272 DEBUG:boto:Params: {} 2024-04-09T22:07:04.272 DEBUG:boto:Token: None 2024-04-09T22:07:04.272 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/?versions 2024-04-09T22:07:04.272 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao= 2024-04-09T22:07:04.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jpY555p9olX9bHufkrF5S8UV3ao=', 'Content-Length': '0'} 2024-04-09T22:07:04.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082291505a5a43524-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.275 DEBUG:boto:b'xryqqk-19999falseobjnulltrue2024-04-09T22:06:48.948Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:07:04.275 DEBUG:boto:path=/xryqqk-19/obj 2024-04-09T22:07:04.275 DEBUG:boto:auth_path=/xryqqk-19/obj 2024-04-09T22:07:04.275 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.275 DEBUG:boto:Path: /xryqqk-19/obj 2024-04-09T22:07:04.275 DEBUG:boto:Data: 2024-04-09T22:07:04.276 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.276 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.276 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.276 DEBUG:boto:Params: {} 2024-04-09T22:07:04.276 DEBUG:boto:Token: None 2024-04-09T22:07:04.276 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/obj 2024-04-09T22:07:04.276 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RhtfDZGRUNmqm4RlHBtPqTm66qw= 2024-04-09T22:07:04.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RhtfDZGRUNmqm4RlHBtPqTm66qw=', 'Content-Length': '0'} 2024-04-09T22:07:04.277 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000524d1ad93f086a63-006615bc08-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.278 DEBUG:boto:path=/xryqqk-19/obj 2024-04-09T22:07:04.278 DEBUG:boto:auth_path=/xryqqk-19/obj 2024-04-09T22:07:04.278 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.278 DEBUG:boto:Path: /xryqqk-19/obj 2024-04-09T22:07:04.278 DEBUG:boto:Data: 2024-04-09T22:07:04.278 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.278 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.278 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.278 DEBUG:boto:Params: {} 2024-04-09T22:07:04.278 DEBUG:boto:Token: None 2024-04-09T22:07:04.279 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-19/obj 2024-04-09T22:07:04.279 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RhtfDZGRUNmqm4RlHBtPqTm66qw= 2024-04-09T22:07:04.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RhtfDZGRUNmqm4RlHBtPqTm66qw=', 'Content-Length': '0'} 2024-04-09T22:07:04.280 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000037e3a39ba486dbb8-006615bc08-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.280 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-19 zones={a2, a1} 2024-04-09T22:07:04.280 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-20 2024-04-09T22:07:04.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:07:04.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:07:04.281 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-20 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:07:04.338 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:04.338 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:04.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.373+0000 7faff5a10780 1 bucket sync caught up with source: 2024-04-09T22:07:04.375 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000001.12.14, , ] 2024-04-09T22:07:04.375 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000001.12.14, , ] 2024-04-09T22:07:04.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.373+0000 7faff5a10780 0 bucket checkpoint complete 2024-04-09T22:07:04.387 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-20 2024-04-09T22:07:04.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:07:04.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:07:04.388 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-20 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:07:04.444 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:04.444 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:04.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.478+0000 7efda5ce1780 1 bucket sync caught up with source: 2024-04-09T22:07:04.479 INFO:teuthology.orchestra.run.smithi063.stderr: local status: [, , , , , , , , 00000000003.34.7, , ] 2024-04-09T22:07:04.480 INFO:teuthology.orchestra.run.smithi063.stderr: remote markers: [, , , , , , , , 00000000003.34.7, , ] 2024-04-09T22:07:04.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:04.478+0000 7efda5ce1780 0 bucket checkpoint complete 2024-04-09T22:07:04.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=xryqqk-20 zones={a2, a1} 2024-04-09T22:07:04.493 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:07:04.493 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:07:04.493 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.494 DEBUG:boto:Path: /xryqqk-20/ 2024-04-09T22:07:04.494 DEBUG:boto:Data: 2024-04-09T22:07:04.494 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.494 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.494 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.494 DEBUG:boto:Params: {} 2024-04-09T22:07:04.494 DEBUG:boto:Token: None 2024-04-09T22:07:04.494 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/ 2024-04-09T22:07:04.494 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/z/5JwzZBOaAbbS8VVjRTPtlanU= 2024-04-09T22:07:04.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/z/5JwzZBOaAbbS8VVjRTPtlanU=', 'Content-Length': '0'} 2024-04-09T22:07:04.497 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6e92c0130ef4c56-006615bc08-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.497 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:07:04.498 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:07:04.498 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.498 DEBUG:boto:Path: /xryqqk-20/ 2024-04-09T22:07:04.502 DEBUG:boto:Data: 2024-04-09T22:07:04.502 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.502 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.502 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.502 DEBUG:boto:Params: {} 2024-04-09T22:07:04.503 DEBUG:boto:Token: None 2024-04-09T22:07:04.503 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/ 2024-04-09T22:07:04.503 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/z/5JwzZBOaAbbS8VVjRTPtlanU= 2024-04-09T22:07:04.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/z/5JwzZBOaAbbS8VVjRTPtlanU=', 'Content-Length': '0'} 2024-04-09T22:07:04.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9f489a2e32204a4-006615bc08-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.505 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-09T22:07:04.505 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:07:04.506 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:07:04.506 DEBUG:boto:path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.506 DEBUG:boto:auth_path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.506 DEBUG:boto:Method: GET 2024-04-09T22:07:04.506 DEBUG:boto:Path: /xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.506 DEBUG:boto:Data: 2024-04-09T22:07:04.506 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.506 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.506 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.506 DEBUG:boto:Params: {} 2024-04-09T22:07:04.507 DEBUG:boto:Token: None 2024-04-09T22:07:04.507 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/?versions 2024-04-09T22:07:04.507 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ= 2024-04-09T22:07:04.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec0e979233cb4a78-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.509 DEBUG:boto:b'xryqqk-20999falseobjnulltrue2024-04-09T22:06:48.972Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:07:04.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-09T22:07:04.510 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:07:04.510 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:07:04.510 DEBUG:boto:path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.510 DEBUG:boto:auth_path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.510 DEBUG:boto:Method: GET 2024-04-09T22:07:04.510 DEBUG:boto:Path: /xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.511 DEBUG:boto:Data: 2024-04-09T22:07:04.511 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.511 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.511 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.511 DEBUG:boto:Params: {} 2024-04-09T22:07:04.511 DEBUG:boto:Token: None 2024-04-09T22:07:04.511 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/?versions 2024-04-09T22:07:04.511 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ= 2024-04-09T22:07:04.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007afc118589d6277b-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.514 DEBUG:boto:b'xryqqk-20999falseobjnulltrue2024-04-09T22:06:48.972Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-09T22:07:04.514 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:07:04.514 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:07:04.514 DEBUG:boto:path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.514 DEBUG:boto:auth_path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.514 DEBUG:boto:Method: GET 2024-04-09T22:07:04.515 DEBUG:boto:Path: /xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.515 DEBUG:boto:Data: 2024-04-09T22:07:04.515 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.515 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.515 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.515 DEBUG:boto:Params: {} 2024-04-09T22:07:04.515 DEBUG:boto:Token: None 2024-04-09T22:07:04.515 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/?versions 2024-04-09T22:07:04.515 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ= 2024-04-09T22:07:04.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d14fc1d55141af67-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.517 DEBUG:boto:b'xryqqk-20999falseobjnulltrue2024-04-09T22:06:48.972Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.518 DEBUG:boto:path=/xryqqk-20/ 2024-04-09T22:07:04.518 DEBUG:boto:auth_path=/xryqqk-20/ 2024-04-09T22:07:04.518 DEBUG:boto:path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.518 DEBUG:boto:auth_path=/xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.518 DEBUG:boto:Method: GET 2024-04-09T22:07:04.518 DEBUG:boto:Path: /xryqqk-20/?versions&max-keys=999 2024-04-09T22:07:04.518 DEBUG:boto:Data: 2024-04-09T22:07:04.518 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.518 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.519 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.519 DEBUG:boto:Params: {} 2024-04-09T22:07:04.519 DEBUG:boto:Token: None 2024-04-09T22:07:04.519 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/?versions 2024-04-09T22:07:04.519 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ= 2024-04-09T22:07:04.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nx8DTO8zTqRdtaxhsPmOj53WzxQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4a8bb9950c9317e-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.521 DEBUG:boto:b'xryqqk-20999falseobjnulltrue2024-04-09T22:06:48.972Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-09T22:07:04.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-09T22:07:04.522 DEBUG:boto:path=/xryqqk-20/obj 2024-04-09T22:07:04.522 DEBUG:boto:auth_path=/xryqqk-20/obj 2024-04-09T22:07:04.522 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.522 DEBUG:boto:Path: /xryqqk-20/obj 2024-04-09T22:07:04.522 DEBUG:boto:Data: 2024-04-09T22:07:04.522 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.522 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.522 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.523 DEBUG:boto:Params: {} 2024-04-09T22:07:04.523 DEBUG:boto:Token: None 2024-04-09T22:07:04.523 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/obj 2024-04-09T22:07:04.523 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7ZbtI6zETIUlYS0NBAw7Pked7b8= 2024-04-09T22:07:04.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ZbtI6zETIUlYS0NBAw7Pked7b8=', 'Content-Length': '0'} 2024-04-09T22:07:04.525 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006d20e6eee1b0c644-006615bc08-1112-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.525 DEBUG:boto:path=/xryqqk-20/obj 2024-04-09T22:07:04.526 DEBUG:boto:auth_path=/xryqqk-20/obj 2024-04-09T22:07:04.526 DEBUG:boto:Method: HEAD 2024-04-09T22:07:04.526 DEBUG:boto:Path: /xryqqk-20/obj 2024-04-09T22:07:04.526 DEBUG:boto:Data: 2024-04-09T22:07:04.526 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.526 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.526 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.526 DEBUG:boto:Params: {} 2024-04-09T22:07:04.526 DEBUG:boto:Token: None 2024-04-09T22:07:04.527 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-20/obj 2024-04-09T22:07:04.527 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7ZbtI6zETIUlYS0NBAw7Pked7b8= 2024-04-09T22:07:04.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ZbtI6zETIUlYS0NBAw7Pked7b8=', 'Content-Length': '0'} 2024-04-09T22:07:04.528 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000acfc04bbf74850f8-006615bc08-1110-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.528 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=xryqqk-20 zones={a2, a1} 2024-04-09T22:07:04.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-04-09T22:07:04.530 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-21 2024-04-09T22:07:04.530 DEBUG:boto:path=/xryqqk-21/ 2024-04-09T22:07:04.530 DEBUG:boto:auth_path=/xryqqk-21/ 2024-04-09T22:07:04.530 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.530 DEBUG:boto:Path: /xryqqk-21/ 2024-04-09T22:07:04.530 DEBUG:boto:Data: 2024-04-09T22:07:04.530 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.530 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.531 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.531 DEBUG:boto:Params: {} 2024-04-09T22:07:04.531 DEBUG:boto:Token: None 2024-04-09T22:07:04.531 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-21/ 2024-04-09T22:07:04.531 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tlq/n0SWzQBIH+Pf/IBH/6LGm9c= 2024-04-09T22:07:04.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tlq/n0SWzQBIH+Pf/IBH/6LGm9c=', 'Content-Length': '0'} 2024-04-09T22:07:04.544 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e3dd7a9433c2e6a5-006615bc08-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.544 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-22 2024-04-09T22:07:04.544 DEBUG:boto:path=/xryqqk-22/ 2024-04-09T22:07:04.544 DEBUG:boto:auth_path=/xryqqk-22/ 2024-04-09T22:07:04.544 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.544 DEBUG:boto:Path: /xryqqk-22/ 2024-04-09T22:07:04.544 DEBUG:boto:Data: 2024-04-09T22:07:04.545 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.545 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.545 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.545 DEBUG:boto:Params: {} 2024-04-09T22:07:04.545 DEBUG:boto:Token: None 2024-04-09T22:07:04.545 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-22/ 2024-04-09T22:07:04.545 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4LhRViacCoC1MA/sk/cnhnPh0CE= 2024-04-09T22:07:04.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4LhRViacCoC1MA/sk/cnhnPh0CE=', 'Content-Length': '0'} 2024-04-09T22:07:04.567 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000039a1326fa568a2c9-006615bc08-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.568 DEBUG:boto:Using access key provided by client. 2024-04-09T22:07:04.568 DEBUG:boto:Using secret key provided by client. 2024-04-09T22:07:04.568 DEBUG:boto:Using access key provided by client. 2024-04-09T22:07:04.568 DEBUG:boto:Using secret key provided by client. 2024-04-09T22:07:04.569 DEBUG:boto:Using access key provided by client. 2024-04-09T22:07:04.569 DEBUG:boto:Using secret key provided by client. 2024-04-09T22:07:04.569 DEBUG:boto:Using access key provided by client. 2024-04-09T22:07:04.569 DEBUG:boto:Using secret key provided by client. 2024-04-09T22:07:04.570 INFO:rgw_multi.tests:create bucket zone=b1 name=xryqqk-23 2024-04-09T22:07:04.570 DEBUG:boto:path=/xryqqk-23/ 2024-04-09T22:07:04.570 DEBUG:boto:auth_path=/xryqqk-23/ 2024-04-09T22:07:04.570 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.570 DEBUG:boto:Path: /xryqqk-23/ 2024-04-09T22:07:04.570 DEBUG:boto:Data: 2024-04-09T22:07:04.570 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.570 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.570 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.570 DEBUG:boto:Params: {} 2024-04-09T22:07:04.571 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:07:04.571 DEBUG:boto:Token: None 2024-04-09T22:07:04.571 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-23/ 2024-04-09T22:07:04.571 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UTiEYv1PmcWFKZb5w56FpHbLd8o= 2024-04-09T22:07:04.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UTiEYv1PmcWFKZb5w56FpHbLd8o=', 'Content-Length': '0'} 2024-04-09T22:07:04.606 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bcf0d51b5aad7b71-006615bc08-10fd-b1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.606 INFO:rgw_multi.tests:create bucket zone=b2 name=xryqqk-24 2024-04-09T22:07:04.606 DEBUG:boto:path=/xryqqk-24/ 2024-04-09T22:07:04.606 DEBUG:boto:auth_path=/xryqqk-24/ 2024-04-09T22:07:04.606 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.606 DEBUG:boto:Path: /xryqqk-24/ 2024-04-09T22:07:04.606 DEBUG:boto:Data: 2024-04-09T22:07:04.606 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.607 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.607 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.607 DEBUG:boto:Params: {} 2024-04-09T22:07:04.607 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:07:04.607 DEBUG:boto:Token: None 2024-04-09T22:07:04.608 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-24/ 2024-04-09T22:07:04.608 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uENdOIFgPUz8EaYpDvSOShBT3GI= 2024-04-09T22:07:04.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uENdOIFgPUz8EaYpDvSOShBT3GI=', 'Content-Length': '0'} 2024-04-09T22:07:04.636 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000075a59a873ff3c296-006615bc08-1103-b2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.636 DEBUG:boto:path=/xryqqk-21/ 2024-04-09T22:07:04.637 DEBUG:boto:auth_path=/xryqqk-21/ 2024-04-09T22:07:04.637 DEBUG:boto:path=/xryqqk-21/?versioning 2024-04-09T22:07:04.637 DEBUG:boto:auth_path=/xryqqk-21/?versioning 2024-04-09T22:07:04.637 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.637 DEBUG:boto:Path: /xryqqk-21/?versioning 2024-04-09T22:07:04.637 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:07:04.637 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.637 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.637 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.637 DEBUG:boto:Params: {} 2024-04-09T22:07:04.638 DEBUG:boto:Token: None 2024-04-09T22:07:04.638 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-21/?versioning 2024-04-09T22:07:04.638 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VUiU8ozMSi0rQlRBWsq7ZCls8jM= 2024-04-09T22:07:04.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VUiU8ozMSi0rQlRBWsq7ZCls8jM=', 'Content-Length': '228'} 2024-04-09T22:07:04.643 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1caaf523c50faf8-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.643 DEBUG:boto:path=/xryqqk-21/ 2024-04-09T22:07:04.643 DEBUG:boto:auth_path=/xryqqk-21/ 2024-04-09T22:07:04.643 DEBUG:boto:path=/xryqqk-21/?versioning 2024-04-09T22:07:04.643 DEBUG:boto:auth_path=/xryqqk-21/?versioning 2024-04-09T22:07:04.643 DEBUG:boto:Method: GET 2024-04-09T22:07:04.643 DEBUG:boto:Path: /xryqqk-21/?versioning 2024-04-09T22:07:04.643 DEBUG:boto:Data: 2024-04-09T22:07:04.643 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.643 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.644 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.644 DEBUG:boto:Params: {} 2024-04-09T22:07:04.644 DEBUG:boto:Token: None 2024-04-09T22:07:04.644 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-21/?versioning 2024-04-09T22:07:04.644 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UPZOY/DjbrIt0lTCMok92JmT54w= 2024-04-09T22:07:04.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UPZOY/DjbrIt0lTCMok92JmT54w=', 'Content-Length': '0'} 2024-04-09T22:07:04.645 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000028e9277b229a07d0-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.645 DEBUG:boto:EnabledDisabled 2024-04-09T22:07:04.646 DEBUG:boto:path=/xryqqk-22/ 2024-04-09T22:07:04.646 DEBUG:boto:auth_path=/xryqqk-22/ 2024-04-09T22:07:04.646 DEBUG:boto:path=/xryqqk-22/?versioning 2024-04-09T22:07:04.646 DEBUG:boto:auth_path=/xryqqk-22/?versioning 2024-04-09T22:07:04.646 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.646 DEBUG:boto:Path: /xryqqk-22/?versioning 2024-04-09T22:07:04.646 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:07:04.646 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.646 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.647 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.647 DEBUG:boto:Params: {} 2024-04-09T22:07:04.647 DEBUG:boto:Token: None 2024-04-09T22:07:04.647 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-22/?versioning 2024-04-09T22:07:04.647 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iI8n3aGeoAd3QXMZxvatAG7LL0s= 2024-04-09T22:07:04.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iI8n3aGeoAd3QXMZxvatAG7LL0s=', 'Content-Length': '228'} 2024-04-09T22:07:04.654 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a71d89dd6c18eb22-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.654 DEBUG:boto:path=/xryqqk-22/ 2024-04-09T22:07:04.654 DEBUG:boto:auth_path=/xryqqk-22/ 2024-04-09T22:07:04.655 DEBUG:boto:path=/xryqqk-22/?versioning 2024-04-09T22:07:04.655 DEBUG:boto:auth_path=/xryqqk-22/?versioning 2024-04-09T22:07:04.655 DEBUG:boto:Method: GET 2024-04-09T22:07:04.655 DEBUG:boto:Path: /xryqqk-22/?versioning 2024-04-09T22:07:04.655 DEBUG:boto:Data: 2024-04-09T22:07:04.655 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.655 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.655 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.655 DEBUG:boto:Params: {} 2024-04-09T22:07:04.655 DEBUG:boto:Token: None 2024-04-09T22:07:04.655 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-22/?versioning 2024-04-09T22:07:04.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IzmyreAClpPITsFAbC2/r+StFnA= 2024-04-09T22:07:04.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IzmyreAClpPITsFAbC2/r+StFnA=', 'Content-Length': '0'} 2024-04-09T22:07:04.657 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf5418a756efa85c-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.657 DEBUG:boto:EnabledDisabled 2024-04-09T22:07:04.657 DEBUG:boto:path=/xryqqk-23/ 2024-04-09T22:07:04.657 DEBUG:boto:auth_path=/xryqqk-23/ 2024-04-09T22:07:04.657 DEBUG:boto:path=/xryqqk-23/?versioning 2024-04-09T22:07:04.658 DEBUG:boto:auth_path=/xryqqk-23/?versioning 2024-04-09T22:07:04.658 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.658 DEBUG:boto:Path: /xryqqk-23/?versioning 2024-04-09T22:07:04.658 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:07:04.658 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.658 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.658 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.658 DEBUG:boto:Params: {} 2024-04-09T22:07:04.658 DEBUG:boto:Token: None 2024-04-09T22:07:04.658 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-23/?versioning 2024-04-09T22:07:04.658 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hpmmJ/5XmS5f2CX8UC4bxxAB2ZM= 2024-04-09T22:07:04.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hpmmJ/5XmS5f2CX8UC4bxxAB2ZM=', 'Content-Length': '228'} 2024-04-09T22:07:04.667 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000972eda922d668c77-006615bc08-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.667 DEBUG:boto:path=/xryqqk-23/ 2024-04-09T22:07:04.667 DEBUG:boto:auth_path=/xryqqk-23/ 2024-04-09T22:07:04.667 DEBUG:boto:path=/xryqqk-23/?versioning 2024-04-09T22:07:04.667 DEBUG:boto:auth_path=/xryqqk-23/?versioning 2024-04-09T22:07:04.667 DEBUG:boto:Method: GET 2024-04-09T22:07:04.667 DEBUG:boto:Path: /xryqqk-23/?versioning 2024-04-09T22:07:04.667 DEBUG:boto:Data: 2024-04-09T22:07:04.667 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.667 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.667 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.668 DEBUG:boto:Params: {} 2024-04-09T22:07:04.668 DEBUG:boto:Token: None 2024-04-09T22:07:04.668 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-23/?versioning 2024-04-09T22:07:04.668 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6+arUIhwAmc/UPEAdXG7rHjxxcs= 2024-04-09T22:07:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6+arUIhwAmc/UPEAdXG7rHjxxcs=', 'Content-Length': '0'} 2024-04-09T22:07:04.669 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ef423ff00fe982b-006615bc08-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.669 DEBUG:boto:EnabledDisabled 2024-04-09T22:07:04.669 DEBUG:boto:path=/xryqqk-24/ 2024-04-09T22:07:04.669 DEBUG:boto:auth_path=/xryqqk-24/ 2024-04-09T22:07:04.669 DEBUG:boto:path=/xryqqk-24/?versioning 2024-04-09T22:07:04.669 DEBUG:boto:auth_path=/xryqqk-24/?versioning 2024-04-09T22:07:04.670 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.670 DEBUG:boto:Path: /xryqqk-24/?versioning 2024-04-09T22:07:04.670 DEBUG:boto:Data: Enabled Disabled 2024-04-09T22:07:04.670 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.670 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.670 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.670 DEBUG:boto:Params: {} 2024-04-09T22:07:04.670 DEBUG:boto:Token: None 2024-04-09T22:07:04.670 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-24/?versioning 2024-04-09T22:07:04.670 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8VxnU6tcdgY68g4umqFBUjOkGms= 2024-04-09T22:07:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8VxnU6tcdgY68g4umqFBUjOkGms=', 'Content-Length': '228'} 2024-04-09T22:07:04.678 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f4c54b7bb6ac130d-006615bc08-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.678 DEBUG:boto:path=/xryqqk-24/ 2024-04-09T22:07:04.678 DEBUG:boto:auth_path=/xryqqk-24/ 2024-04-09T22:07:04.679 DEBUG:boto:path=/xryqqk-24/?versioning 2024-04-09T22:07:04.679 DEBUG:boto:auth_path=/xryqqk-24/?versioning 2024-04-09T22:07:04.679 DEBUG:boto:Method: GET 2024-04-09T22:07:04.679 DEBUG:boto:Path: /xryqqk-24/?versioning 2024-04-09T22:07:04.679 DEBUG:boto:Data: 2024-04-09T22:07:04.679 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.679 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.679 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.679 DEBUG:boto:Params: {} 2024-04-09T22:07:04.679 DEBUG:boto:Token: None 2024-04-09T22:07:04.679 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-24/?versioning 2024-04-09T22:07:04.680 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:03VP7Mx8vE6181UrnWYC+yaCEsI= 2024-04-09T22:07:04.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:03VP7Mx8vE6181UrnWYC+yaCEsI=', 'Content-Length': '0'} 2024-04-09T22:07:04.681 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cac259050fbeaae9-006615bc08-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.681 DEBUG:boto:EnabledDisabled 2024-04-09T22:07:04.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-04-09T22:07:04.682 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-25 2024-04-09T22:07:04.682 DEBUG:boto:path=/xryqqk-25/ 2024-04-09T22:07:04.682 DEBUG:boto:auth_path=/xryqqk-25/ 2024-04-09T22:07:04.682 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.682 DEBUG:boto:Path: /xryqqk-25/ 2024-04-09T22:07:04.682 DEBUG:boto:Data: 2024-04-09T22:07:04.682 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.682 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.683 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.683 DEBUG:boto:Params: {} 2024-04-09T22:07:04.683 DEBUG:boto:Token: None 2024-04-09T22:07:04.683 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-25/ 2024-04-09T22:07:04.683 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vqQ1EWUXguKpuTP7vsBIfdDn4hE= 2024-04-09T22:07:04.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vqQ1EWUXguKpuTP7vsBIfdDn4hE=', 'Content-Length': '0'} 2024-04-09T22:07:04.696 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2e11ed347dc9cd8-006615bc08-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.696 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-26 2024-04-09T22:07:04.696 DEBUG:boto:path=/xryqqk-26/ 2024-04-09T22:07:04.696 DEBUG:boto:auth_path=/xryqqk-26/ 2024-04-09T22:07:04.696 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.696 DEBUG:boto:Path: /xryqqk-26/ 2024-04-09T22:07:04.696 DEBUG:boto:Data: 2024-04-09T22:07:04.696 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.696 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.696 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.697 DEBUG:boto:Params: {} 2024-04-09T22:07:04.697 DEBUG:boto:Token: None 2024-04-09T22:07:04.697 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-26/ 2024-04-09T22:07:04.697 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:FPrwiNVZ1qz8Fp6mgNtdATKRfSQ= 2024-04-09T22:07:04.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FPrwiNVZ1qz8Fp6mgNtdATKRfSQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.718 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003051ab5ac24bce2a-006615bc08-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.718 INFO:rgw_multi.tests:create bucket zone=b1 name=xryqqk-27 2024-04-09T22:07:04.718 DEBUG:boto:path=/xryqqk-27/ 2024-04-09T22:07:04.718 DEBUG:boto:auth_path=/xryqqk-27/ 2024-04-09T22:07:04.718 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.718 DEBUG:boto:Path: /xryqqk-27/ 2024-04-09T22:07:04.718 DEBUG:boto:Data: 2024-04-09T22:07:04.718 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.719 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.719 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.719 DEBUG:boto:Params: {} 2024-04-09T22:07:04.719 DEBUG:boto:Token: None 2024-04-09T22:07:04.719 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-27/ 2024-04-09T22:07:04.719 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5yGyGZlzqlXp8SUzkhQ0GPWoFr8= 2024-04-09T22:07:04.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5yGyGZlzqlXp8SUzkhQ0GPWoFr8=', 'Content-Length': '0'} 2024-04-09T22:07:04.743 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002f2f3a58ce831c6c-006615bc08-10fd-b1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.744 INFO:rgw_multi.tests:create bucket zone=b2 name=xryqqk-28 2024-04-09T22:07:04.744 DEBUG:boto:path=/xryqqk-28/ 2024-04-09T22:07:04.744 DEBUG:boto:auth_path=/xryqqk-28/ 2024-04-09T22:07:04.744 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.744 DEBUG:boto:Path: /xryqqk-28/ 2024-04-09T22:07:04.744 DEBUG:boto:Data: 2024-04-09T22:07:04.744 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.744 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.744 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.744 DEBUG:boto:Params: {} 2024-04-09T22:07:04.745 DEBUG:boto:Token: None 2024-04-09T22:07:04.745 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-28/ 2024-04-09T22:07:04.745 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sIoqVHu2eSzGmwZFBbhwDCXb+cw= 2024-04-09T22:07:04.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sIoqVHu2eSzGmwZFBbhwDCXb+cw=', 'Content-Length': '0'} 2024-04-09T22:07:04.792 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cab5346dd2389090-006615bc08-1103-b2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.792 DEBUG:boto:path=/xryqqk-25/ 2024-04-09T22:07:04.792 DEBUG:boto:auth_path=/xryqqk-25/ 2024-04-09T22:07:04.793 DEBUG:boto:path=/xryqqk-25/?acl 2024-04-09T22:07:04.793 DEBUG:boto:auth_path=/xryqqk-25/?acl 2024-04-09T22:07:04.793 DEBUG:boto:Method: GET 2024-04-09T22:07:04.793 DEBUG:boto:Path: /xryqqk-25/?acl 2024-04-09T22:07:04.793 DEBUG:boto:Data: 2024-04-09T22:07:04.793 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.793 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.793 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.793 DEBUG:boto:Params: {} 2024-04-09T22:07:04.793 DEBUG:boto:Token: None 2024-04-09T22:07:04.794 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-25/?acl 2024-04-09T22:07:04.794 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8A8LOkaDImVANBYACchf9c4H7BQ= 2024-04-09T22:07:04.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8A8LOkaDImVANBYACchf9c4H7BQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.795 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007c3b4c1841165041-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.795 DEBUG:boto:path=/xryqqk-25/ 2024-04-09T22:07:04.795 DEBUG:boto:auth_path=/xryqqk-25/ 2024-04-09T22:07:04.795 DEBUG:boto:path=/xryqqk-25/?acl 2024-04-09T22:07:04.796 DEBUG:boto:auth_path=/xryqqk-25/?acl 2024-04-09T22:07:04.796 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.796 DEBUG:boto:Path: /xryqqk-25/?acl 2024-04-09T22:07:04.796 DEBUG:boto:Data: 2024-04-09T22:07:04.796 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T22:07:04.796 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.796 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.796 DEBUG:boto:Params: {} 2024-04-09T22:07:04.796 DEBUG:boto:Token: None 2024-04-09T22:07:04.796 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT x-amz-acl:public-read /xryqqk-25/?acl 2024-04-09T22:07:04.797 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pW3dCuqOEGYCV9lu6CS6oC5uCHc= 2024-04-09T22:07:04.797 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pW3dCuqOEGYCV9lu6CS6oC5uCHc=', 'Content-Length': '0'} 2024-04-09T22:07:04.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001aec22ad1e012beb-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.804 DEBUG:boto:path=/xryqqk-25/ 2024-04-09T22:07:04.804 DEBUG:boto:auth_path=/xryqqk-25/ 2024-04-09T22:07:04.804 DEBUG:boto:path=/xryqqk-25/?acl 2024-04-09T22:07:04.805 DEBUG:boto:auth_path=/xryqqk-25/?acl 2024-04-09T22:07:04.805 DEBUG:boto:Method: GET 2024-04-09T22:07:04.805 DEBUG:boto:Path: /xryqqk-25/?acl 2024-04-09T22:07:04.805 DEBUG:boto:Data: 2024-04-09T22:07:04.805 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.805 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.805 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.805 DEBUG:boto:Params: {} 2024-04-09T22:07:04.805 DEBUG:boto:Token: None 2024-04-09T22:07:04.805 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-25/?acl 2024-04-09T22:07:04.806 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8A8LOkaDImVANBYACchf9c4H7BQ= 2024-04-09T22:07:04.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8A8LOkaDImVANBYACchf9c4H7BQ=', 'Content-Length': '0'} 2024-04-09T22:07:04.807 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002c918f58de6eabbd-006615bc08-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.807 DEBUG:boto:path=/xryqqk-26/ 2024-04-09T22:07:04.807 DEBUG:boto:auth_path=/xryqqk-26/ 2024-04-09T22:07:04.807 DEBUG:boto:path=/xryqqk-26/?acl 2024-04-09T22:07:04.807 DEBUG:boto:auth_path=/xryqqk-26/?acl 2024-04-09T22:07:04.807 DEBUG:boto:Method: GET 2024-04-09T22:07:04.807 DEBUG:boto:Path: /xryqqk-26/?acl 2024-04-09T22:07:04.807 DEBUG:boto:Data: 2024-04-09T22:07:04.808 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.808 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.808 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.808 DEBUG:boto:Params: {} 2024-04-09T22:07:04.808 DEBUG:boto:Token: None 2024-04-09T22:07:04.808 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-26/?acl 2024-04-09T22:07:04.808 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t3+30zn6e7YgZuH3nwBw9Ngw3a8= 2024-04-09T22:07:04.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t3+30zn6e7YgZuH3nwBw9Ngw3a8=', 'Content-Length': '0'} 2024-04-09T22:07:04.809 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007dad6fc81457b29d-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.809 DEBUG:boto:path=/xryqqk-26/ 2024-04-09T22:07:04.810 DEBUG:boto:auth_path=/xryqqk-26/ 2024-04-09T22:07:04.810 DEBUG:boto:path=/xryqqk-26/?acl 2024-04-09T22:07:04.810 DEBUG:boto:auth_path=/xryqqk-26/?acl 2024-04-09T22:07:04.810 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.810 DEBUG:boto:Path: /xryqqk-26/?acl 2024-04-09T22:07:04.810 DEBUG:boto:Data: 2024-04-09T22:07:04.810 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T22:07:04.810 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.810 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.810 DEBUG:boto:Params: {} 2024-04-09T22:07:04.811 DEBUG:boto:Token: None 2024-04-09T22:07:04.811 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT x-amz-acl:public-read /xryqqk-26/?acl 2024-04-09T22:07:04.811 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qihpsdJvUp19rDyu3d/mdYtvbdU= 2024-04-09T22:07:04.811 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qihpsdJvUp19rDyu3d/mdYtvbdU=', 'Content-Length': '0'} 2024-04-09T22:07:04.818 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003cdbbab708105f9e-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.818 DEBUG:boto:path=/xryqqk-26/ 2024-04-09T22:07:04.818 DEBUG:boto:auth_path=/xryqqk-26/ 2024-04-09T22:07:04.818 DEBUG:boto:path=/xryqqk-26/?acl 2024-04-09T22:07:04.818 DEBUG:boto:auth_path=/xryqqk-26/?acl 2024-04-09T22:07:04.819 DEBUG:boto:Method: GET 2024-04-09T22:07:04.819 DEBUG:boto:Path: /xryqqk-26/?acl 2024-04-09T22:07:04.819 DEBUG:boto:Data: 2024-04-09T22:07:04.819 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.819 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.819 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.819 DEBUG:boto:Params: {} 2024-04-09T22:07:04.819 DEBUG:boto:Token: None 2024-04-09T22:07:04.819 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-26/?acl 2024-04-09T22:07:04.819 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t3+30zn6e7YgZuH3nwBw9Ngw3a8= 2024-04-09T22:07:04.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t3+30zn6e7YgZuH3nwBw9Ngw3a8=', 'Content-Length': '0'} 2024-04-09T22:07:04.820 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6f957b83bfa31d4-006615bc08-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.867 DEBUG:boto:path=/xryqqk-27/ 2024-04-09T22:07:04.867 DEBUG:boto:auth_path=/xryqqk-27/ 2024-04-09T22:07:04.867 DEBUG:boto:path=/xryqqk-27/?acl 2024-04-09T22:07:04.867 DEBUG:boto:auth_path=/xryqqk-27/?acl 2024-04-09T22:07:04.867 DEBUG:boto:Method: GET 2024-04-09T22:07:04.867 DEBUG:boto:Path: /xryqqk-27/?acl 2024-04-09T22:07:04.867 DEBUG:boto:Data: 2024-04-09T22:07:04.867 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.867 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.868 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.868 DEBUG:boto:Params: {} 2024-04-09T22:07:04.868 DEBUG:boto:Token: None 2024-04-09T22:07:04.868 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-27/?acl 2024-04-09T22:07:04.868 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3AfVZgrOpQu79HhYuLrWbLgrLR4= 2024-04-09T22:07:04.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3AfVZgrOpQu79HhYuLrWbLgrLR4=', 'Content-Length': '0'} 2024-04-09T22:07:04.869 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd3f2b4c476e1d0b-006615bc08-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.870 DEBUG:boto:path=/xryqqk-27/ 2024-04-09T22:07:04.870 DEBUG:boto:auth_path=/xryqqk-27/ 2024-04-09T22:07:04.870 DEBUG:boto:path=/xryqqk-27/?acl 2024-04-09T22:07:04.870 DEBUG:boto:auth_path=/xryqqk-27/?acl 2024-04-09T22:07:04.870 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.870 DEBUG:boto:Path: /xryqqk-27/?acl 2024-04-09T22:07:04.871 DEBUG:boto:Data: 2024-04-09T22:07:04.871 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T22:07:04.871 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.871 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.871 DEBUG:boto:Params: {} 2024-04-09T22:07:04.871 DEBUG:boto:Token: None 2024-04-09T22:07:04.871 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT x-amz-acl:public-read /xryqqk-27/?acl 2024-04-09T22:07:04.871 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qwzE1jvEu9dUWnJigglPC0voE4E= 2024-04-09T22:07:04.871 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qwzE1jvEu9dUWnJigglPC0voE4E=', 'Content-Length': '0'} 2024-04-09T22:07:04.880 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000363b0d7564f5bf63-006615bc08-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.880 DEBUG:boto:path=/xryqqk-27/ 2024-04-09T22:07:04.880 DEBUG:boto:auth_path=/xryqqk-27/ 2024-04-09T22:07:04.880 DEBUG:boto:path=/xryqqk-27/?acl 2024-04-09T22:07:04.880 DEBUG:boto:auth_path=/xryqqk-27/?acl 2024-04-09T22:07:04.880 DEBUG:boto:Method: GET 2024-04-09T22:07:04.881 DEBUG:boto:Path: /xryqqk-27/?acl 2024-04-09T22:07:04.881 DEBUG:boto:Data: 2024-04-09T22:07:04.881 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.881 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.881 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.881 DEBUG:boto:Params: {} 2024-04-09T22:07:04.881 DEBUG:boto:Token: None 2024-04-09T22:07:04.881 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-27/?acl 2024-04-09T22:07:04.881 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3AfVZgrOpQu79HhYuLrWbLgrLR4= 2024-04-09T22:07:04.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3AfVZgrOpQu79HhYuLrWbLgrLR4=', 'Content-Length': '0'} 2024-04-09T22:07:04.883 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba2367f7ec72c1ce-006615bc08-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.883 DEBUG:boto:path=/xryqqk-28/ 2024-04-09T22:07:04.883 DEBUG:boto:auth_path=/xryqqk-28/ 2024-04-09T22:07:04.883 DEBUG:boto:path=/xryqqk-28/?acl 2024-04-09T22:07:04.883 DEBUG:boto:auth_path=/xryqqk-28/?acl 2024-04-09T22:07:04.883 DEBUG:boto:Method: GET 2024-04-09T22:07:04.884 DEBUG:boto:Path: /xryqqk-28/?acl 2024-04-09T22:07:04.884 DEBUG:boto:Data: 2024-04-09T22:07:04.884 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.884 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.884 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.884 DEBUG:boto:Params: {} 2024-04-09T22:07:04.884 DEBUG:boto:Token: None 2024-04-09T22:07:04.884 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-28/?acl 2024-04-09T22:07:04.884 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:G+LlinAZ9cnYCi8cFvMYsBoEJAY= 2024-04-09T22:07:04.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:G+LlinAZ9cnYCi8cFvMYsBoEJAY=', 'Content-Length': '0'} 2024-04-09T22:07:04.886 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ffb6719046e4cbd-006615bc08-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.886 DEBUG:boto:path=/xryqqk-28/ 2024-04-09T22:07:04.886 DEBUG:boto:auth_path=/xryqqk-28/ 2024-04-09T22:07:04.886 DEBUG:boto:path=/xryqqk-28/?acl 2024-04-09T22:07:04.886 DEBUG:boto:auth_path=/xryqqk-28/?acl 2024-04-09T22:07:04.886 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.886 DEBUG:boto:Path: /xryqqk-28/?acl 2024-04-09T22:07:04.887 DEBUG:boto:Data: 2024-04-09T22:07:04.887 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-09T22:07:04.887 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.887 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.887 DEBUG:boto:Params: {} 2024-04-09T22:07:04.887 DEBUG:boto:Token: None 2024-04-09T22:07:04.887 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT x-amz-acl:public-read /xryqqk-28/?acl 2024-04-09T22:07:04.887 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0mJ4JnGAc2Uy8B6wDrFuIykGI84= 2024-04-09T22:07:04.887 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0mJ4JnGAc2Uy8B6wDrFuIykGI84=', 'Content-Length': '0'} 2024-04-09T22:07:04.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009178d123dff4412a-006615bc08-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.895 DEBUG:boto:path=/xryqqk-28/ 2024-04-09T22:07:04.896 DEBUG:boto:auth_path=/xryqqk-28/ 2024-04-09T22:07:04.896 DEBUG:boto:path=/xryqqk-28/?acl 2024-04-09T22:07:04.896 DEBUG:boto:auth_path=/xryqqk-28/?acl 2024-04-09T22:07:04.896 DEBUG:boto:Method: GET 2024-04-09T22:07:04.896 DEBUG:boto:Path: /xryqqk-28/?acl 2024-04-09T22:07:04.896 DEBUG:boto:Data: 2024-04-09T22:07:04.896 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.896 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:04.896 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.896 DEBUG:boto:Params: {} 2024-04-09T22:07:04.897 DEBUG:boto:Token: None 2024-04-09T22:07:04.897 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-28/?acl 2024-04-09T22:07:04.897 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:G+LlinAZ9cnYCi8cFvMYsBoEJAY= 2024-04-09T22:07:04.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:G+LlinAZ9cnYCi8cFvMYsBoEJAY=', 'Content-Length': '0'} 2024-04-09T22:07:04.898 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008db7914d666d05cf-006615bc08-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-04-09T22:07:04.939 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-29 2024-04-09T22:07:04.940 DEBUG:boto:path=/xryqqk-29/ 2024-04-09T22:07:04.940 DEBUG:boto:auth_path=/xryqqk-29/ 2024-04-09T22:07:04.940 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.940 DEBUG:boto:Path: /xryqqk-29/ 2024-04-09T22:07:04.940 DEBUG:boto:Data: 2024-04-09T22:07:04.940 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.940 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:04.940 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.940 DEBUG:boto:Params: {} 2024-04-09T22:07:04.941 DEBUG:boto:Token: None 2024-04-09T22:07:04.941 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-29/ 2024-04-09T22:07:04.941 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ytrIsm9TQaT42iZBa/yN3b2TUvA= 2024-04-09T22:07:04.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ytrIsm9TQaT42iZBa/yN3b2TUvA=', 'Content-Length': '0'} 2024-04-09T22:07:04.954 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d67c55599fa0e4dd-006615bc08-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.954 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-30 2024-04-09T22:07:04.954 DEBUG:boto:path=/xryqqk-30/ 2024-04-09T22:07:04.954 DEBUG:boto:auth_path=/xryqqk-30/ 2024-04-09T22:07:04.954 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.955 DEBUG:boto:Path: /xryqqk-30/ 2024-04-09T22:07:04.955 DEBUG:boto:Data: 2024-04-09T22:07:04.955 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.955 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:04.955 DEBUG:boto:Port: 8001 2024-04-09T22:07:04.955 DEBUG:boto:Params: {} 2024-04-09T22:07:04.955 DEBUG:boto:Token: None 2024-04-09T22:07:04.955 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-30/ 2024-04-09T22:07:04.956 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:APR3GvGuFfVGHkaTrFOpXOOy8/Q= 2024-04-09T22:07:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:APR3GvGuFfVGHkaTrFOpXOOy8/Q=', 'Content-Length': '0'} 2024-04-09T22:07:04.993 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000065b6217599c67d09-006615bc08-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:04.994 INFO:rgw_multi.tests:create bucket zone=b1 name=xryqqk-31 2024-04-09T22:07:04.994 DEBUG:boto:path=/xryqqk-31/ 2024-04-09T22:07:04.994 DEBUG:boto:auth_path=/xryqqk-31/ 2024-04-09T22:07:04.994 DEBUG:boto:Method: PUT 2024-04-09T22:07:04.994 DEBUG:boto:Path: /xryqqk-31/ 2024-04-09T22:07:04.994 DEBUG:boto:Data: 2024-04-09T22:07:04.994 DEBUG:boto:Headers: {} 2024-04-09T22:07:04.994 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:04.995 DEBUG:boto:Port: 8000 2024-04-09T22:07:04.995 DEBUG:boto:Params: {} 2024-04-09T22:07:04.995 DEBUG:boto:Token: None 2024-04-09T22:07:04.995 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:04 GMT /xryqqk-31/ 2024-04-09T22:07:04.995 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PBQ35H2gn4AGecUc+EiKJlblTRw= 2024-04-09T22:07:04.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PBQ35H2gn4AGecUc+EiKJlblTRw=', 'Content-Length': '0'} 2024-04-09T22:07:05.020 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000002cc25874637743-006615bc08-10fd-b1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:05.020 INFO:rgw_multi.tests:create bucket zone=b2 name=xryqqk-32 2024-04-09T22:07:05.020 DEBUG:boto:path=/xryqqk-32/ 2024-04-09T22:07:05.021 DEBUG:boto:auth_path=/xryqqk-32/ 2024-04-09T22:07:05.021 DEBUG:boto:Method: PUT 2024-04-09T22:07:05.021 DEBUG:boto:Path: /xryqqk-32/ 2024-04-09T22:07:05.021 DEBUG:boto:Data: 2024-04-09T22:07:05.021 DEBUG:boto:Headers: {} 2024-04-09T22:07:05.021 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:05.021 DEBUG:boto:Port: 8001 2024-04-09T22:07:05.021 DEBUG:boto:Params: {} 2024-04-09T22:07:05.021 DEBUG:boto:Token: None 2024-04-09T22:07:05.022 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:05 GMT /xryqqk-32/ 2024-04-09T22:07:05.022 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ayGfGOfgnrtd8HijjE2TJHE/C+A= 2024-04-09T22:07:05.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ayGfGOfgnrtd8HijjE2TJHE/C+A=', 'Content-Length': '0'} 2024-04-09T22:07:05.045 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d6628ca9e67b1388-006615bc09-1103-b2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:05.046 DEBUG:boto:path=/xryqqk-29/ 2024-04-09T22:07:05.046 DEBUG:boto:auth_path=/xryqqk-29/ 2024-04-09T22:07:05.046 DEBUG:boto:path=/xryqqk-29/?cors 2024-04-09T22:07:05.046 DEBUG:boto:auth_path=/xryqqk-29/?cors 2024-04-09T22:07:05.046 DEBUG:boto:Method: PUT 2024-04-09T22:07:05.046 DEBUG:boto:Path: /xryqqk-29/?cors 2024-04-09T22:07:05.046 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-09T22:07:05.046 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-09T22:07:05.046 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:05.046 DEBUG:boto:Port: 8000 2024-04-09T22:07:05.046 DEBUG:boto:Params: {} 2024-04-09T22:07:05.047 DEBUG:boto:Token: None 2024-04-09T22:07:05.047 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 09 Apr 2024 22:07:05 GMT /xryqqk-29/?cors 2024-04-09T22:07:05.047 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4RQXIUxr1DuXqcJb7xVsA+3YVM4= 2024-04-09T22:07:05.047 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4RQXIUxr1DuXqcJb7xVsA+3YVM4=', 'Content-Length': '218'} 2024-04-09T22:07:05.052 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000038fdaa97479f940c-006615bc09-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:05.052 DEBUG:boto:path=/xryqqk-29/ 2024-04-09T22:07:05.052 DEBUG:boto:auth_path=/xryqqk-29/ 2024-04-09T22:07:05.052 DEBUG:boto:path=/xryqqk-29/?cors 2024-04-09T22:07:05.052 DEBUG:boto:auth_path=/xryqqk-29/?cors 2024-04-09T22:07:05.052 DEBUG:boto:Method: GET 2024-04-09T22:07:05.052 DEBUG:boto:Path: /xryqqk-29/?cors 2024-04-09T22:07:05.052 DEBUG:boto:Data: 2024-04-09T22:07:05.052 DEBUG:boto:Headers: {} 2024-04-09T22:07:05.052 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:05.052 DEBUG:boto:Port: 8000 2024-04-09T22:07:05.053 DEBUG:boto:Params: {} 2024-04-09T22:07:05.053 DEBUG:boto:Token: None 2024-04-09T22:07:05.053 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:05 GMT /xryqqk-29/?cors 2024-04-09T22:07:05.053 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:NfyJluv7PIUpgLxk0L04teX8muM= 2024-04-09T22:07:05.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NfyJluv7PIUpgLxk0L04teX8muM=', 'Content-Length': '0'} 2024-04-09T22:07:05.054 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000089dff1dff383e3b7-006615bc09-1110-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 09 Apr 2024 22:07:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:05.094 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-09T22:07:05.095 DEBUG:boto:path=/xryqqk-30/ 2024-04-09T22:07:05.095 DEBUG:boto:auth_path=/xryqqk-30/ 2024-04-09T22:07:05.095 DEBUG:boto:path=/xryqqk-30/?cors 2024-04-09T22:07:05.095 DEBUG:boto:auth_path=/xryqqk-30/?cors 2024-04-09T22:07:05.095 DEBUG:boto:Method: PUT 2024-04-09T22:07:05.095 DEBUG:boto:Path: /xryqqk-30/?cors 2024-04-09T22:07:05.095 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-09T22:07:05.095 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-09T22:07:05.095 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:05.096 DEBUG:boto:Port: 8001 2024-04-09T22:07:05.096 DEBUG:boto:Params: {} 2024-04-09T22:07:05.096 DEBUG:boto:Token: None 2024-04-09T22:07:05.096 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 09 Apr 2024 22:07:05 GMT /xryqqk-30/?cors 2024-04-09T22:07:05.096 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t8VOg3af+VHLediPsLTVhl8FywE= 2024-04-09T22:07:05.096 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:05 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t8VOg3af+VHLediPsLTVhl8FywE=', 'Content-Length': '218'} 2024-04-09T22:07:06.105 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5e8ddb63ec74209-006615bc09-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:06.105 DEBUG:boto:path=/xryqqk-30/ 2024-04-09T22:07:06.105 DEBUG:boto:auth_path=/xryqqk-30/ 2024-04-09T22:07:06.105 DEBUG:boto:path=/xryqqk-30/?cors 2024-04-09T22:07:06.105 DEBUG:boto:auth_path=/xryqqk-30/?cors 2024-04-09T22:07:06.105 DEBUG:boto:Method: GET 2024-04-09T22:07:06.106 DEBUG:boto:Path: /xryqqk-30/?cors 2024-04-09T22:07:06.106 DEBUG:boto:Data: 2024-04-09T22:07:06.106 DEBUG:boto:Headers: {} 2024-04-09T22:07:06.106 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:06.106 DEBUG:boto:Port: 8001 2024-04-09T22:07:06.106 DEBUG:boto:Params: {} 2024-04-09T22:07:06.106 DEBUG:boto:Token: None 2024-04-09T22:07:06.106 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:06 GMT /xryqqk-30/?cors 2024-04-09T22:07:06.107 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wR1fdxKEA2vRuFGwfJb3T9vdaK4= 2024-04-09T22:07:06.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:06 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wR1fdxKEA2vRuFGwfJb3T9vdaK4=', 'Content-Length': '0'} 2024-04-09T22:07:06.108 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5e9c01a12e2937b-006615bc0a-1112-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 09 Apr 2024 22:07:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:06.108 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-09T22:07:06.108 DEBUG:boto:path=/xryqqk-31/ 2024-04-09T22:07:06.109 DEBUG:boto:auth_path=/xryqqk-31/ 2024-04-09T22:07:06.109 DEBUG:boto:path=/xryqqk-31/?cors 2024-04-09T22:07:06.109 DEBUG:boto:auth_path=/xryqqk-31/?cors 2024-04-09T22:07:06.109 DEBUG:boto:Method: PUT 2024-04-09T22:07:06.109 DEBUG:boto:Path: /xryqqk-31/?cors 2024-04-09T22:07:06.109 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-09T22:07:06.109 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-09T22:07:06.109 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:06.109 DEBUG:boto:Port: 8000 2024-04-09T22:07:06.109 DEBUG:boto:Params: {} 2024-04-09T22:07:06.110 DEBUG:boto:Token: None 2024-04-09T22:07:06.110 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 09 Apr 2024 22:07:06 GMT /xryqqk-31/?cors 2024-04-09T22:07:06.110 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4IbMOszQ+Or98dfkUrnpBg0PrY8= 2024-04-09T22:07:06.110 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:06 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4IbMOszQ+Or98dfkUrnpBg0PrY8=', 'Content-Length': '218'} 2024-04-09T22:07:07.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000cb60eadd62396fc-006615bc0a-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:07.120 DEBUG:boto:path=/xryqqk-31/ 2024-04-09T22:07:07.120 DEBUG:boto:auth_path=/xryqqk-31/ 2024-04-09T22:07:07.120 DEBUG:boto:path=/xryqqk-31/?cors 2024-04-09T22:07:07.120 DEBUG:boto:auth_path=/xryqqk-31/?cors 2024-04-09T22:07:07.120 DEBUG:boto:Method: GET 2024-04-09T22:07:07.120 DEBUG:boto:Path: /xryqqk-31/?cors 2024-04-09T22:07:07.120 DEBUG:boto:Data: 2024-04-09T22:07:07.120 DEBUG:boto:Headers: {} 2024-04-09T22:07:07.121 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:07:07.121 DEBUG:boto:Port: 8000 2024-04-09T22:07:07.121 DEBUG:boto:Params: {} 2024-04-09T22:07:07.121 DEBUG:boto:Token: None 2024-04-09T22:07:07.121 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:07 GMT /xryqqk-31/?cors 2024-04-09T22:07:07.121 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XWpk4qRrteIkYjs3kUBMBUYsJYo= 2024-04-09T22:07:07.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XWpk4qRrteIkYjs3kUBMBUYsJYo=', 'Content-Length': '0'} 2024-04-09T22:07:07.122 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c2586b81ec95afc4-006615bc0b-10fd-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 09 Apr 2024 22:07:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:07.166 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-09T22:07:07.167 DEBUG:boto:path=/xryqqk-32/ 2024-04-09T22:07:07.167 DEBUG:boto:auth_path=/xryqqk-32/ 2024-04-09T22:07:07.167 DEBUG:boto:path=/xryqqk-32/?cors 2024-04-09T22:07:07.167 DEBUG:boto:auth_path=/xryqqk-32/?cors 2024-04-09T22:07:07.167 DEBUG:boto:Method: PUT 2024-04-09T22:07:07.167 DEBUG:boto:Path: /xryqqk-32/?cors 2024-04-09T22:07:07.168 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-09T22:07:07.168 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-09T22:07:07.168 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:07.168 DEBUG:boto:Port: 8001 2024-04-09T22:07:07.168 DEBUG:boto:Params: {} 2024-04-09T22:07:07.168 DEBUG:boto:Token: None 2024-04-09T22:07:07.168 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Tue, 09 Apr 2024 22:07:07 GMT /xryqqk-32/?cors 2024-04-09T22:07:07.168 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fLKY0T4oqZM2AJt07eIWjjAkDt8= 2024-04-09T22:07:07.169 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fLKY0T4oqZM2AJt07eIWjjAkDt8=', 'Content-Length': '218'} 2024-04-09T22:07:08.178 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000008879b8d5bcc6827-006615bc0b-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:08.179 DEBUG:boto:path=/xryqqk-32/ 2024-04-09T22:07:08.179 DEBUG:boto:auth_path=/xryqqk-32/ 2024-04-09T22:07:08.179 DEBUG:boto:path=/xryqqk-32/?cors 2024-04-09T22:07:08.179 DEBUG:boto:auth_path=/xryqqk-32/?cors 2024-04-09T22:07:08.179 DEBUG:boto:Method: GET 2024-04-09T22:07:08.179 DEBUG:boto:Path: /xryqqk-32/?cors 2024-04-09T22:07:08.179 DEBUG:boto:Data: 2024-04-09T22:07:08.180 DEBUG:boto:Headers: {} 2024-04-09T22:07:08.180 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:07:08.180 DEBUG:boto:Port: 8001 2024-04-09T22:07:08.180 DEBUG:boto:Params: {} 2024-04-09T22:07:08.180 DEBUG:boto:Token: None 2024-04-09T22:07:08.180 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:07:08 GMT /xryqqk-32/?cors 2024-04-09T22:07:08.180 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:azBt1dk1X8NS/afwGIbtNtq6RQE= 2024-04-09T22:07:08.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:azBt1dk1X8NS/afwGIbtNtq6RQE=', 'Content-Length': '0'} 2024-04-09T22:07:08.181 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000229946d5ef8e60ff-006615bc0c-1103-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Tue, 09 Apr 2024 22:07:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:08.182 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-09T22:07:08.182 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-04-09T22:07:08.183 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-33 2024-04-09T22:07:08.183 DEBUG:boto:path=/xryqqk-33/ 2024-04-09T22:07:08.183 DEBUG:boto:auth_path=/xryqqk-33/ 2024-04-09T22:07:08.183 DEBUG:boto:Method: PUT 2024-04-09T22:07:08.183 DEBUG:boto:Path: /xryqqk-33/ 2024-04-09T22:07:08.183 DEBUG:boto:Data: 2024-04-09T22:07:08.183 DEBUG:boto:Headers: {} 2024-04-09T22:07:08.183 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:07:08.184 DEBUG:boto:Port: 8000 2024-04-09T22:07:08.184 DEBUG:boto:Params: {} 2024-04-09T22:07:08.184 DEBUG:boto:Token: None 2024-04-09T22:07:08.184 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:08 GMT /xryqqk-33/ 2024-04-09T22:07:08.184 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:t0Dyv1juq9uPR1bpCu0iIykDnpE= 2024-04-09T22:07:08.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t0Dyv1juq9uPR1bpCu0iIykDnpE=', 'Content-Length': '0'} 2024-04-09T22:07:08.199 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001cc64436c6383c10-006615bc0c-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:08.199 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-34 2024-04-09T22:07:08.200 DEBUG:boto:path=/xryqqk-34/ 2024-04-09T22:07:08.200 DEBUG:boto:auth_path=/xryqqk-34/ 2024-04-09T22:07:08.200 DEBUG:boto:Method: PUT 2024-04-09T22:07:08.200 DEBUG:boto:Path: /xryqqk-34/ 2024-04-09T22:07:08.200 DEBUG:boto:Data: 2024-04-09T22:07:08.200 DEBUG:boto:Headers: {} 2024-04-09T22:07:08.200 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:07:08.200 DEBUG:boto:Port: 8001 2024-04-09T22:07:08.200 DEBUG:boto:Params: {} 2024-04-09T22:07:08.200 DEBUG:boto:Token: None 2024-04-09T22:07:08.201 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:07:08 GMT /xryqqk-34/ 2024-04-09T22:07:08.201 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ljVltV4r3La8cU0Ve4AQDctjmck= 2024-04-09T22:07:08.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ljVltV4r3La8cU0Ve4AQDctjmck=', 'Content-Length': '0'} 2024-04-09T22:07:08.223 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000074b0f3b647de9631-006615bc0c-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:07:08.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:08.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:08.224 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:08.273 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:08.273 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:08.303 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:07:08.315 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:07:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:07:08.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700428.214372_408.1", 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:07:08.214372Z" 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:07:08.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:08.317 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:07:08.318 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:08.318 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:07:08.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:08.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:08.318 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:08.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:08.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:08.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:08.459+0000 7f34d1156780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:08.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:08.460+0000 7f34d1156780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:08.474 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.586841_338.1", 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.586841Z", 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:08Z" 2024-04-09T22:07:08.480 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:08.481 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.586841_338.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.586841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:08Z" } 2024-04-09T22:07:08.481 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.586841_338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.586841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:08.481 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:08.481 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.586841_338.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:08.481 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.586841_338.1 2024-04-09T22:07:13.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:13.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:13.483 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:13.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:13.555 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:13.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:13.635+0000 7f5922fd0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:13.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:13.637+0000 7f5922fd0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:13.650 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:13.650 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:13.650 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:13.650 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:13.650 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:13.650 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:13.651 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:13.652 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.555171_334.1", 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:13.653 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:13.654 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:13.655 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:13Z" 2024-04-09T22:07:13.656 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:13.656 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.555171_334.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.555171Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:13Z" } 2024-04-09T22:07:13.657 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.555171_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.555171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:13.657 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:13.657 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.555171_334.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:13.657 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.555171_334.1 2024-04-09T22:07:18.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:18.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:18.658 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:18.709 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:18.709 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:18.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:18.736+0000 7f649554f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:18.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:18.737+0000 7f649554f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:18.750 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:18.750 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:18.750 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:18.750 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:18.750 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:18.750 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:18.751 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:18.752 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.539236_330.1", 2024-04-09T22:07:18.753 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:18.754 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:18.755 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:18.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:18.756 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:18Z" 2024-04-09T22:07:18.756 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:18.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.539236_330.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.539236Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:18Z" } 2024-04-09T22:07:18.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.539236_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.539236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:18.757 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:18.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.539236_330.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:18.757 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.539236_330.1 2024-04-09T22:07:23.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:23.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:23.759 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:23.808 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:23.808 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:23.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:23.844+0000 7fc122569780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:23.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:23.845+0000 7fc122569780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:23.858 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:23.858 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:23.858 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:23.859 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:23.860 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:23.861 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:23.862 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:23.863 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:23.864 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:23.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:23.864 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:23Z" 2024-04-09T22:07:23.864 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:23.865 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:23Z" } 2024-04-09T22:07:23.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:23.865 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:23.865 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:23.865 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.661401_349.1 2024-04-09T22:07:28.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:28.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:28.867 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:28.916 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:28.916 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:28.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:28.962+0000 7f80ca4fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:28.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:28.973+0000 7f80ca4fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:29.018 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:29.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:29.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.539236_330.1", 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:29.021 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:29.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:28Z" 2024-04-09T22:07:29.023 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:29.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.539236_330.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.539236Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:28Z" } 2024-04-09T22:07:29.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.539236_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.539236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:29.024 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:29.024 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.539236_330.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:29.025 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.539236_330.1 2024-04-09T22:07:34.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:34.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:34.026 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:34.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:34.103+0000 7fc0f3cd5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:34.104+0000 7fc0f3cd5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:34.117 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:34.117 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:34.117 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:34.118 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:34.119 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:34.120 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.542449_332.1", 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.542449Z", 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:34.121 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:34.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:34.123 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:34.123 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:34.123 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:34.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:34.123 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:34Z" 2024-04-09T22:07:34.123 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:34.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.542449_332.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.542449Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:34Z" } 2024-04-09T22:07:34.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.542449_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.542449Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:34.124 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:34.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.542449_332.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:34.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.542449_332.1 2024-04-09T22:07:39.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:39.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:39.126 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:39.176 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:39.176 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:39.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:39.205+0000 7fe3e9fbc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:39.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:39.206+0000 7fe3e9fbc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:39.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:39.220 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:39.221 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.622748_342.1", 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:39.222 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.622748Z", 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:39.223 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:39.224 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:39.225 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:39.225 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:39.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:39.225 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:39Z" 2024-04-09T22:07:39.225 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:39.226 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.622748_342.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.622748Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:39Z" } 2024-04-09T22:07:39.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.622748_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.622748Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:39.226 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:39.226 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.622748_342.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:39.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.622748_342.1 2024-04-09T22:07:44.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:44.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:44.227 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:44.279 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:44.280 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:44.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:44.308+0000 7f69e53c6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:44.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:44.309+0000 7f69e53c6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:44.322 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:44.323 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:44.324 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:44.325 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.584285_337.1", 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:44.326 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:44.327 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:44.328 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:44.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:44.328 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:44Z" 2024-04-09T22:07:44.328 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:44.329 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.584285_337.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.584285Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:44Z" } 2024-04-09T22:07:44.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.584285_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.584285Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:44.329 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:44.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.584285_337.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:44.329 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.584285_337.1 2024-04-09T22:07:49.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:49.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:49.331 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:49.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:49.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:49.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:49.455+0000 7f77f50c6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:49.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:49.456+0000 7f77f50c6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:49.471 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:49.471 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:49.472 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:49.472 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:49.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:49.472 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:49.473 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:49.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:49.473 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:49.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:49.474 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:49.474 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:49.474 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:49.474 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:49.474 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:49.475 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:49.475 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:49.475 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:49.475 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:49.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:49.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:49.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:49.476 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:49.476 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:49.476 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:49.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:49.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:49.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:49.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:49.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:49.478 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:49.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:49.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:49.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:49.478 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:49.479 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:49.479 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:49.479 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.584285_337.1", 2024-04-09T22:07:49.479 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:49.479 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:49.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:49.480 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-09T22:07:49.480 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:49.480 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:49.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:49.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:49.481 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:49.481 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:49.481 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:49.481 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:49.481 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:49.482 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:49.482 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:49.482 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:49.482 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:49.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:49.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:49.483 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:49.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:49.483 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:49.483 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:49.483 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:49.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:49.484 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:49Z" 2024-04-09T22:07:49.484 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:49.485 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.584285_337.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.584285Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:49Z" } 2024-04-09T22:07:49.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.584285_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.584285Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:49.486 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:49.486 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.584285_337.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:49.486 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.584285_337.1 2024-04-09T22:07:54.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:54.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:54.488 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:54.566 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:54.566 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:54.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:54.659+0000 7fada877d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:54.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:54.661+0000 7fada877d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:54.679 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:54.679 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:54.679 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:54.679 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:54.679 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:54.679 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:54.680 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:54.681 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.555171_334.1", 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:54.682 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:54Z" 2024-04-09T22:07:54.684 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:54.685 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.555171_334.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.555171Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:54Z" } 2024-04-09T22:07:54.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.555171_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.555171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:54.686 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:54.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.555171_334.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:54.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.555171_334.1 2024-04-09T22:07:59.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:59.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:07:59.687 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:07:59.738 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:07:59.739 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:07:59.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:59.768+0000 7fe4fcbcd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:59.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:07:59.769+0000 7fe4fcbcd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:07:59.782 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:07:59.782 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:07:59.782 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:59.783 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:07:59.784 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.625978_344.1", 2024-04-09T22:07:59.785 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.625978Z", 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:07:59.786 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:07:59.787 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:07:59.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:07:59.788 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:07:59Z" 2024-04-09T22:07:59.788 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:07:59.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.625978_344.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.625978Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:07:59Z" } 2024-04-09T22:07:59.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.625978_344.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.625978Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:07:59.789 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:59.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.625978_344.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:07:59.789 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.625978_344.1 2024-04-09T22:08:04.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:04.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:04.790 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:04.840 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:04.840 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:04.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:04.869+0000 7fd49a131780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:04.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:04.870+0000 7fd49a131780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:04.882 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:04.882 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:04.882 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:04.882 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:04.882 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:04.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:04.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.639793_345.1", 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:04.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.639793Z", 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:04.886 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:04Z" 2024-04-09T22:08:04.887 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:04.888 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.639793_345.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.639793Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:04Z" } 2024-04-09T22:08:04.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.639793_345.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.639793Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:04.889 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:04.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.639793_345.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:04.889 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.639793_345.1 2024-04-09T22:08:09.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:09.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:09.891 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:09.949 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:09.949 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:09.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:09.984+0000 7f22ec8e4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:09.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:09.985+0000 7f22ec8e4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:09.999 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:09.999 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:10.000 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:10.001 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.555171_334.1", 2024-04-09T22:08:10.002 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:10.003 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:10.004 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:10.005 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:09Z" 2024-04-09T22:08:10.005 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:10.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.555171_334.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.555171Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:09Z" } 2024-04-09T22:08:10.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.555171_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.555171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:10.006 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:10.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.555171_334.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:10.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.555171_334.1 2024-04-09T22:08:15.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:15.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:15.008 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:15.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:15.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:15.190+0000 7f0009262780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:15.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:15.194+0000 7f0009262780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:15.212 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:15.213 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:15.214 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:15.215 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.558529_336.1", 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.558529Z", 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:15.216 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:15.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:15.218 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:15Z" 2024-04-09T22:08:15.218 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:15.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.558529_336.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.558529Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:15Z" } 2024-04-09T22:08:15.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.558529_336.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.558529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:15.219 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:15.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.558529_336.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:15.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.558529_336.1 2024-04-09T22:08:20.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:20.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:20.221 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:20.271 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:20.271 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:20.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:20.299+0000 7f1293bb9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:20.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:20.300+0000 7f1293bb9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:20.314 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:20.314 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:20.314 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:20.314 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:20.314 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:20.315 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:20.316 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:20.317 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.675822_352.1", 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:20.318 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:20.319 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:20.320 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:20.320 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:20.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:20.320 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:20Z" 2024-04-09T22:08:20.320 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:20.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.675822_352.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.675822Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:20Z" } 2024-04-09T22:08:20.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:20.321 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:20.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:20.322 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.675822_352.1 2024-04-09T22:08:25.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:25.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:25.323 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:25.375 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:25.375 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:25.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:25.403+0000 7f0e0c9ec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:25.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:25.404+0000 7f0e0c9ec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:25.417 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:25.418 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:25.419 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.540137_331.1", 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.540137Z", 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:25.420 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:25.421 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:25Z" 2024-04-09T22:08:25.422 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:25.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.540137_331.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.540137Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:25Z" } 2024-04-09T22:08:25.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.540137_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.540137Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:25.423 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:25.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.540137_331.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:25.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.540137_331.1 2024-04-09T22:08:30.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:30.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:30.425 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:30.477 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:30.477 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:30.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:30.505+0000 7f339b945780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:30.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:30.506+0000 7f339b945780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:30.519 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:30.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:30.521 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.558529_336.1", 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.558529Z", 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:30.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:30.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:30Z" 2024-04-09T22:08:30.524 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:30.525 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.558529_336.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.558529Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:30Z" } 2024-04-09T22:08:30.525 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.558529_336.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.558529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:30.525 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:30.525 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.558529_336.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:30.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.558529_336.1 2024-04-09T22:08:35.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:35.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:35.528 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:35.586 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:35.587 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:35.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:35.619+0000 7fab881ef780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:35.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:35.620+0000 7fab881ef780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:35.634 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:35.634 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:35.635 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:35.636 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:35.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.539236_330.1", 2024-04-09T22:08:35.638 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:35.639 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:35.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:35.641 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:35Z" 2024-04-09T22:08:35.641 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:35.642 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.539236_330.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.539236Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:35Z" } 2024-04-09T22:08:35.642 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.539236_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.539236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:35.642 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:35.642 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.539236_330.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:35.642 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.539236_330.1 2024-04-09T22:08:40.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:40.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:40.643 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:40.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:40.732+0000 7fdaafae3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:40.733+0000 7fdaafae3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.620202_341.1", 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.620202Z", 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:40.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:40Z" 2024-04-09T22:08:40.753 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:40.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.620202_341.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.620202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:40Z" } 2024-04-09T22:08:40.754 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.620202_341.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.620202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:40.754 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:40.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.620202_341.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:40.755 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.620202_341.1 2024-04-09T22:08:45.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:45.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:45.756 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:45.835 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:45.835 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:45.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:45.932+0000 7fe7128c3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:45.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:45.934+0000 7fe7128c3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:45.965 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:45.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:45.967 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.584285_337.1", 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:45.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:45.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:45.970 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:45.970 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:45.970 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:45.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:45.970 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:45Z" 2024-04-09T22:08:45.970 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:45.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.584285_337.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.584285Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:45Z" } 2024-04-09T22:08:45.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.584285_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.584285Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:45.971 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:45.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.584285_337.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:45.971 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.584285_337.1 2024-04-09T22:08:50.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:50.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:50.973 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:51.022 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:51.023 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:51.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:51.050+0000 7f687c824780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:51.051+0000 7f687c824780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:51.064 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:51.064 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:51.064 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:51.065 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:51.066 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:51.067 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.556117_335.1", 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.556117Z", 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:51.068 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:51.069 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:51.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:51.070 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:51Z" 2024-04-09T22:08:51.070 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:51.071 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.556117_335.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.556117Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:51Z" } 2024-04-09T22:08:51.071 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.556117_335.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.556117Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:51.071 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:51.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.556117_335.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:51.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.556117_335.1 2024-04-09T22:08:56.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:56.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:08:56.072 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:08:56.124 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:08:56.124 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:08:56.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:56.153+0000 7fd91e0fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:56.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:08:56.154+0000 7fd91e0fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:08:56.167 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:08:56.167 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:08:56.167 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:08:56.167 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:56.168 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:56.169 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:08:56.170 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:08:56.171 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:08:56.172 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:56.173 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:08:56.173 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:08:56.173 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:08:56.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:08:56.173 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:08:56Z" 2024-04-09T22:08:56.173 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:08:56.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:08:56Z" } 2024-04-09T22:08:56.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:08:56.174 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:56.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:08:56.174 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-09T22:09:01.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:01.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:01.176 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:01.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:01.266+0000 7f0257e34780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:01.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:01.267+0000 7f0257e34780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:01.281 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:01.281 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:01.281 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:01.281 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:01.283 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:01.284 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:01.285 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:01.286 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:01.287 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:01.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:01.287 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:01Z" 2024-04-09T22:09:01.287 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:01.288 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:01Z" } 2024-04-09T22:09:01.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:01.288 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:01.288 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:01.288 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-09T22:09:06.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:06.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:06.289 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:06.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:06.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:06.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:06.481+0000 7fd197416780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:06.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:06.482+0000 7fd197416780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:06.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:06.498 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:06.499 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:06.500 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:06.501 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:06Z" 2024-04-09T22:09:06.502 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:06.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:06Z" } 2024-04-09T22:09:06.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:06.504 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:06.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:06.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-09T22:09:11.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:11.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:11.505 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:11.557 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:11.558 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:11.589+0000 7fb311f41780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:11.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:11.590+0000 7fb311f41780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:11.604 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:11.604 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:11.604 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:11.605 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:11.605 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:11.605 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:11.605 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:11.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:11.606 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:11.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:11.606 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:11.606 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:11.607 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:11.607 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:11.607 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:11.607 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:11.607 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:11.608 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:11.608 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:11.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:11.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:11.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:11.609 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:11.609 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:11.609 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:11.609 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:11.609 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:11.610 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:11.610 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:11.610 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:11.610 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:11.610 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:11.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:11.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:11.611 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:11.611 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:11.611 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:11.611 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:09:11.612 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:11.612 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:11.612 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:11.612 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:09:11.612 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:11.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:11.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:11.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:11.613 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:11.613 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:11.614 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:11.614 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:11.614 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:11.614 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:11.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:11.615 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:11.615 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:11.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:11.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:11.615 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:11.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:11.616 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:11.616 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:11.616 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:11.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:11.617 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:11Z" 2024-04-09T22:09:11.617 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:11.618 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:11Z" } 2024-04-09T22:09:11.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:11.619 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:11.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:11.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-09T22:09:16.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:16.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:16.621 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:16.678 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:16.678 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:16.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:16.709+0000 7fded642a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:16.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:16.711+0000 7fded642a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:16Z" 2024-04-09T22:09:16.730 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:16.731 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:16Z" } 2024-04-09T22:09:16.732 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:16.732 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:16.732 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:16.732 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-09T22:09:21.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:21.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:21.733 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:21.789 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:21.789 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:21.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:21.830+0000 7f7fc112f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:21.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:21.834+0000 7f7fc112f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:21.904 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:21.904 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:21.905 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:21.906 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:21.907 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:21.908 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:21.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:21.910 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:21.910 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:21.910 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:21.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:21.910 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:21Z" 2024-04-09T22:09:21.910 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:21.911 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:21Z" } 2024-04-09T22:09:21.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:21.911 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:21.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:21.912 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-09T22:09:26.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:26.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:26.913 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:26.967 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:26.967 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:26.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:26.996+0000 7fce4f155780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:26.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:26.997+0000 7fce4f155780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:27.011 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:27.011 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:27.011 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:27.011 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:27.012 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:27.013 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:27.015 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:27.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:27.017 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:27.017 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:27.017 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:27.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:27.017 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:26Z" 2024-04-09T22:09:27.017 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:27.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:26Z" } 2024-04-09T22:09:27.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:27.018 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:27.018 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:27.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-09T22:09:32.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:32.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:32.020 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:32.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:32.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:32.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:32.105+0000 7f4fd3f68780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:32.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:32.107+0000 7f4fd3f68780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:32.120 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:32.120 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:32.120 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:32.121 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:32.122 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:32.123 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:32.124 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:32.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:32.126 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:32.126 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:32.126 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:32.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:32.126 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:32Z" 2024-04-09T22:09:32.126 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:32.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:32Z" } 2024-04-09T22:09:32.127 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:32.127 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:32.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:32.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-09T22:09:37.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:37.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:37.129 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:37.185 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:37.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:37.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:37.216+0000 7f41f3b41780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:37.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:37.217+0000 7f41f3b41780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:37.232 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:37.232 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:37.232 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:37.232 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:37.232 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:37.232 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:37.233 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:37.234 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:09:37.235 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:37.236 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:37.237 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:37.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:37.238 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:37Z" 2024-04-09T22:09:37.238 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:37.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:37Z" } 2024-04-09T22:09:37.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:37.239 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:37.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:37.239 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.874323_373.1 2024-04-09T22:09:42.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:42.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:42.241 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:42.295 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:42.296 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:42.324+0000 7fd52f51b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:42.326+0000 7fd52f51b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:42.339 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:42.339 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:42.339 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:42.339 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:42.339 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:42.340 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:42.341 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.691210_354.1", 2024-04-09T22:09:42.342 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.691210Z", 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:42.343 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:42.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:42.345 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:42.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:42.345 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:42Z" 2024-04-09T22:09:42.345 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:42.346 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.691210_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.691210Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:42Z" } 2024-04-09T22:09:42.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.691210_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.691210Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:42.346 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:42.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.691210_354.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:42.346 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.691210_354.1 2024-04-09T22:09:47.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:47.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:47.348 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:47.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:47.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:47.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:47.434+0000 7ffa3da22780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:47.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:47.435+0000 7ffa3da22780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:47.448 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:47.449 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:47.450 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.691210_354.1", 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:47.451 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.691210Z", 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:47.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:47Z" 2024-04-09T22:09:47.453 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:47.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.691210_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.691210Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:47Z" } 2024-04-09T22:09:47.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.691210_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.691210Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:47.455 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:47.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.691210_354.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:47.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.691210_354.1 2024-04-09T22:09:52.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:52.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:52.457 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:52.515 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:52.515 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:52.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:52.547+0000 7f8874792780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:52.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:52.548+0000 7f8874792780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:52.562 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:52.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:52.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:52.565 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.692156_355.1", 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:52.566 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:52.567 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:52.568 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:52.569 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:52.569 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:52.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:52.569 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:52Z" 2024-04-09T22:09:52.569 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:52.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.692156_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.692156Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:52Z" } 2024-04-09T22:09:52.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:52.570 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:52.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:52.571 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.692156_355.1 2024-04-09T22:09:57.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:57.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:09:57.572 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:09:57.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:09:57.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:09:57.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:57.684+0000 7fdbab0ec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:09:57.685+0000 7fdbab0ec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:09:57.699 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:09:57.699 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:57.700 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:57.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:57.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:57.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:09:57.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:09:57Z" 2024-04-09T22:09:57.705 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:09:57.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:09:57Z" } 2024-04-09T22:09:57.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:09:57.706 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:57.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:09:57.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-09T22:10:02.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:02.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:02.708 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:02.760 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:02.760 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:02.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:02.788+0000 7f1502bac780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:02.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:02.789+0000 7f1502bac780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:02.803 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:02.803 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:02.803 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:02.804 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:02.805 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:02.805 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:02.805 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:02.805 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:02.806 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:02.806 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:02.806 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:02.806 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:02.807 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:02.808 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:02.808 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:02.808 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:02.808 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:02.808 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:02.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:02.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:02.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:02.809 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:02.810 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:02.810 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:02.810 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:10:02.810 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:02.810 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:02.811 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:02.811 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:10:02.811 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:02.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:02.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:02.812 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:02.813 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:02.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:02.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:02.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:02.813 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:02.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:02.814 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:02.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:02.814 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:02.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:02.815 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:02Z" 2024-04-09T22:10:02.815 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:02.816 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:02Z" } 2024-04-09T22:10:02.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:02.817 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:02.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:02.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-09T22:10:07.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:07.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:07.819 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:07.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:07.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:07.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:07.911+0000 7ff748698780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:07.912+0000 7ff748698780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:07.927 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:07.928 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:07.929 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:07.930 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:07.931 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:07Z" 2024-04-09T22:10:07.932 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:07.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:07Z" } 2024-04-09T22:10:07.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:07.934 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:07.934 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:07.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-09T22:10:12.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:12.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:12.936 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:12.986 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:12.987 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:13.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:13.015+0000 7fef222c2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:13.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:13.016+0000 7fef222c2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:13.029 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:13.029 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:13.029 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:13.030 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:13.031 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:13.032 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.675822_352.1", 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:13.033 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:13.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:13.035 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:13.035 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:13.035 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:13.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:13.035 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:13Z" 2024-04-09T22:10:13.035 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:13.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.675822_352.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.675822Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:13Z" } 2024-04-09T22:10:13.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:13.036 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:13.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:13.037 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.675822_352.1 2024-04-09T22:10:18.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:18.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:18.038 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:18.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:18.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:18.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:18.132+0000 7f3441762780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:18.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:18.133+0000 7f3441762780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:18.147 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:18.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:18.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.707433_359.1", 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.707433Z", 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:18.150 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:18.151 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:18Z" 2024-04-09T22:10:18.152 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:18.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.707433_359.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.707433Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:18Z" } 2024-04-09T22:10:18.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.707433_359.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.707433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:18.154 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:18.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.707433_359.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:18.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.707433_359.1 2024-04-09T22:10:23.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:23.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:23.155 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:23.214 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:23.214 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:23.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:23.246+0000 7f4bcc07e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:23.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:23.247+0000 7f4bcc07e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:23.261 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:23.262 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:23.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:23.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:23.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:23.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:23Z" 2024-04-09T22:10:23.267 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:23.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:23Z" } 2024-04-09T22:10:23.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:23.268 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:23.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:23.269 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-09T22:10:28.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:28.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:28.270 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:28.384 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:28.384 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:28.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:28.418+0000 7f94866dd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:28.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:28.419+0000 7f94866dd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:28.435 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:28.435 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:28.435 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:28.436 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:28.437 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:28.438 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:28.439 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:28.440 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:28.441 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:28.441 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:28.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:28.441 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:28Z" 2024-04-09T22:10:28.441 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:28.442 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:28Z" } 2024-04-09T22:10:28.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:28.442 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:28.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:28.442 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-09T22:10:33.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:33.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:33.444 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:33.497 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:33.497 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:33.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:33.526+0000 7f3269c09780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:33.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:33.527+0000 7f3269c09780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:33.542 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:33.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:33.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:33.545 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:10:33.546 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:33.547 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:33.548 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:33.549 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:33Z" 2024-04-09T22:10:33.549 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:33.549 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:33Z" } 2024-04-09T22:10:33.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:33.550 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:33.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:33.550 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-09T22:10:38.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:38.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:38.552 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:38.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:38.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:38.632+0000 7f556fb81780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:38.632+0000 7f556fb81780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:38.645 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:38.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:38.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:38Z" 2024-04-09T22:10:38.651 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:38.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:38Z" } 2024-04-09T22:10:38.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:38.653 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:38.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:38.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-09T22:10:43.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:43.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:43.655 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:43.706 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:43.707 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:43.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:43.737+0000 7f9874a04780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:43.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:43.739+0000 7f9874a04780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:43.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:43.753 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:43.754 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:43.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:10:43.756 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:43.757 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:43.758 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:43.758 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:43.758 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:43.758 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:43.758 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:43Z" 2024-04-09T22:10:43.759 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:43.760 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:43Z" } 2024-04-09T22:10:43.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:43.761 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:43.761 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:43.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-09T22:10:48.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:48.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:48.762 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:48.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:48.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:48.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:48.845+0000 7fbbc15e8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:48.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:48.846+0000 7fbbc15e8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:48.861 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:48.861 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:48.861 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:48.861 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:48.861 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:48.861 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:48.862 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:48.863 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:48.863 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:48.863 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:48.863 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:48.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:48.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:48.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:48.864 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:48.864 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:48.864 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:48.864 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:48.865 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:48.865 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:48.865 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:48.865 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:48.865 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:48.866 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:48.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:48.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:48.866 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:48.866 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:48.866 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:48.867 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.673463_351.1", 2024-04-09T22:10:48.867 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:48.867 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:48.867 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:48.867 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-09T22:10:48.867 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:48.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:48.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:48.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:48.868 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:48.868 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:48.868 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:48.869 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:48.869 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:48.869 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:48.869 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:48.869 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:48.869 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:48.870 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:48Z" 2024-04-09T22:10:48.871 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:48.871 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.673463_351.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.673463Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:48Z" } 2024-04-09T22:10:48.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:48.872 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:48.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:48.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.673463_351.1 2024-04-09T22:10:53.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:53.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:53.873 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:53.925 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:54.042 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:53.953+0000 7f6ba274c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:54.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:53.954+0000 7f6ba274c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:54.047 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:54.048 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:54.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:54.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:54.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:53Z" 2024-04-09T22:10:54.052 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:54.052 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:53Z" } 2024-04-09T22:10:54.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:54.053 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:54.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:54.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-09T22:10:59.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:59.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:10:59.054 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:10:59.113 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:10:59.114 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:10:59.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:59.146+0000 7f55d25ea780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:59.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:10:59.147+0000 7f55d25ea780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:10:59.161 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:10:59.161 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:10:59.161 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:10:59.161 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:10:59.161 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:10:59.162 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:10:59.163 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:59.164 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:10:59.165 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:59.166 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:10:59.167 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:10:59.167 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:10:59.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:10:59.167 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:10:59Z" 2024-04-09T22:10:59.167 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:10:59.168 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:10:59Z" } 2024-04-09T22:10:59.168 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:10:59.168 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:59.168 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:10:59.169 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-09T22:11:04.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:04.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:04.170 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:04.221 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:04.221 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:04.250+0000 7f0284b9b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:04.251+0000 7f0284b9b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:04.264 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:04.265 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:04.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:04.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:04.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:04.269 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:04Z" 2024-04-09T22:11:04.270 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:04.271 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:04Z" } 2024-04-09T22:11:04.271 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:04.272 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:04.272 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:04.272 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-09T22:11:09.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:09.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:09.274 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:09.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:09.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:09.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:09.353+0000 7f4095eab780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:09.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:09.354+0000 7f4095eab780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:09.367 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:09.367 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:09.367 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:09.367 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:09.368 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:09.369 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:09.370 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:09.371 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:09.372 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:09.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:09.373 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:09.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:09.373 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:09Z" 2024-04-09T22:11:09.373 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:09.374 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:09Z" } 2024-04-09T22:11:09.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:09.374 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:09.374 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:09.374 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-09T22:11:14.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:14.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:14.376 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:14.433 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:14.434 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:14.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:14.470+0000 7fef2f1b4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:14.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:14.471+0000 7fef2f1b4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:14.486 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:14.486 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:14.486 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:14.486 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:14.486 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:14.486 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:14.487 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:14.488 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.673463_351.1", 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:14.489 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:14.490 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:14.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:14.492 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:14Z" 2024-04-09T22:11:14.492 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:14.492 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.673463_351.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.673463Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:14Z" } 2024-04-09T22:11:14.493 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:14.493 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:14.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:14.493 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.673463_351.1 2024-04-09T22:11:19.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:19.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:19.494 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:19.552 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:19.552 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:19.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:19.586+0000 7f03c0dbc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:19.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:19.587+0000 7f03c0dbc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:19.602 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:19.602 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:19.603 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:19.604 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.692156_355.1", 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:19.605 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:19.606 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:19.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:19.608 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:19Z" 2024-04-09T22:11:19.608 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:19.609 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.692156_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.692156Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:19Z" } 2024-04-09T22:11:19.609 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:19.609 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:19.609 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:19.609 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.692156_355.1 2024-04-09T22:11:24.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:24.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:24.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:24.661 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:24.661 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:24.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:24.691+0000 7f51ea741780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:24.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:24.692+0000 7f51ea741780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:24.704 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:24.704 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:24.704 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:24.705 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:24.706 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:24.707 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:24.708 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:24.709 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:24.710 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:24.710 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:24.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:24.710 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:24Z" 2024-04-09T22:11:24.710 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:24.711 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:24Z" } 2024-04-09T22:11:24.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:24.711 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:24.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:24.712 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-09T22:11:29.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:29.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:29.713 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:29.763 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:29.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:29.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:29.793+0000 7f3610144780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:29.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:29.794+0000 7f3610144780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:29.808 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:29.809 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:29.810 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:29.811 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:29Z" 2024-04-09T22:11:29.813 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:29.814 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:29Z" } 2024-04-09T22:11:29.815 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:29.815 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:29.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:29.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.688829_353.1 2024-04-09T22:11:34.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:34.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:34.816 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:34.874 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:34.874 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:34.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:34.906+0000 7f547bac8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:34.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:34.907+0000 7f547bac8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:34.921 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:34.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:34.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:34.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:34.926 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:34Z" 2024-04-09T22:11:34.927 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:34.928 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:34Z" } 2024-04-09T22:11:34.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:34.929 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:34.929 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:34.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-09T22:11:39.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:39.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:39.930 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:39.989 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:39.989 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:40.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:40.022+0000 7f909ebb2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:40.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:40.023+0000 7f909ebb2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:40.104 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:40.104 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:40.104 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:40.105 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:40.106 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:40.107 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:40.108 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:40.110 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:40.110 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:40.110 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:40.110 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:40.110 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:40Z" 2024-04-09T22:11:40.110 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:40.111 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:40Z" } 2024-04-09T22:11:40.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:40.111 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:40.111 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:40.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.661401_349.1 2024-04-09T22:11:45.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:45.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:45.114 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:45.164 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:45.164 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:45.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:45.193+0000 7f8c4e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:45.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:45.194+0000 7f8c4e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:45.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:45.209 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:45.210 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:45.211 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.730558_363.1", 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:45.212 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:45.213 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:45.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:45.214 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:45Z" 2024-04-09T22:11:45.214 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:45.215 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.730558_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.730558Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:45Z" } 2024-04-09T22:11:45.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:45.215 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:45.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:45.215 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.730558_363.1 2024-04-09T22:11:50.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:50.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:50.217 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:50.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:50.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:50.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:50.295+0000 7f36a6e0d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:50.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:50.296+0000 7f36a6e0d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:50.310 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:50.311 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:50.312 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:50.313 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:50.314 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:50.315 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:50.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:50.316 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:50Z" 2024-04-09T22:11:50.316 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:50.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:50Z" } 2024-04-09T22:11:50.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:50.317 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:50.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:50.317 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.874323_373.1 2024-04-09T22:11:55.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:55.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:11:55.318 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:11:55.376 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:11:55.377 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:11:55.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:55.409+0000 7f4952ad8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:55.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:11:55.410+0000 7f4952ad8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:11:55.424 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:11:55.424 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:11:55.424 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:55.425 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:55.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:55.427 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:55.428 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:11:55.429 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:11:55.430 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:55.431 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:11:55.431 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:11:55.431 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:11:55.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:11:55.431 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:11:55Z" 2024-04-09T22:11:55.431 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:11:55.432 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:11:55Z" } 2024-04-09T22:11:55.432 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:11:55.432 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:55.433 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:11:55.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-09T22:12:00.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:00.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:00.434 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:00.486 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:00.486 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:00.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:00.520+0000 7f1fc6383780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:00.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:00.521+0000 7f1fc6383780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:00.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:00.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:00.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.692156_355.1", 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:00.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:00.539 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:00Z" 2024-04-09T22:12:00.540 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:00.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.692156_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.692156Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:00Z" } 2024-04-09T22:12:00.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:00.542 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:00.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:00.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.692156_355.1 2024-04-09T22:12:05.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:05.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:05.544 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:05.597 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:05.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:05.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:05.626+0000 7ff43592b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:05.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:05.627+0000 7ff43592b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:05.641 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:05.641 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:05.641 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:05.641 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:05.641 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:05.642 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:05.643 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:05.644 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.709706_360.1", 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:05.645 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:05.646 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:05.647 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:05.647 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:05.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:05.647 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:05Z" 2024-04-09T22:12:05.647 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:05.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.709706_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.709706Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:05Z" } 2024-04-09T22:12:05.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:05.648 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:05.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:05.648 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.709706_360.1 2024-04-09T22:12:10.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:10.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:10.650 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:10.706 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:10.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:10.738+0000 7f976202e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:10.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:10.739+0000 7f976202e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:10.753 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:10.753 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:10.754 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:10.755 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:10.756 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:10.757 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:10.758 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:10Z" 2024-04-09T22:12:10.759 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:10.760 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:10Z" } 2024-04-09T22:12:10.760 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:10.760 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:10.760 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:10.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-09T22:12:15.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-04-09T22:12:15.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2024-04-09T22:12:15.767 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-35 2024-04-09T22:12:15.767 DEBUG:boto:path=/xryqqk-35/ 2024-04-09T22:12:15.767 DEBUG:boto:auth_path=/xryqqk-35/ 2024-04-09T22:12:15.767 DEBUG:boto:Method: PUT 2024-04-09T22:12:15.767 DEBUG:boto:Path: /xryqqk-35/ 2024-04-09T22:12:15.768 DEBUG:boto:Data: 2024-04-09T22:12:15.768 DEBUG:boto:Headers: {} 2024-04-09T22:12:15.768 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:12:15.768 DEBUG:boto:Port: 8000 2024-04-09T22:12:15.768 DEBUG:boto:Params: {} 2024-04-09T22:12:15.768 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:12:15.768 DEBUG:boto:Token: None 2024-04-09T22:12:15.769 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:12:15 GMT /xryqqk-35/ 2024-04-09T22:12:15.769 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EjHP2iSI7DNFh6ZZviM1wsdFRck= 2024-04-09T22:12:15.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EjHP2iSI7DNFh6ZZviM1wsdFRck=', 'Content-Length': '0'} 2024-04-09T22:12:15.848 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d7817a620d10f3de-006615bd3f-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:12:15.848 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-36 2024-04-09T22:12:15.848 DEBUG:boto:path=/xryqqk-36/ 2024-04-09T22:12:15.849 DEBUG:boto:auth_path=/xryqqk-36/ 2024-04-09T22:12:15.849 DEBUG:boto:Method: PUT 2024-04-09T22:12:15.849 DEBUG:boto:Path: /xryqqk-36/ 2024-04-09T22:12:15.849 DEBUG:boto:Data: 2024-04-09T22:12:15.849 DEBUG:boto:Headers: {} 2024-04-09T22:12:15.849 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:12:15.849 DEBUG:boto:Port: 8001 2024-04-09T22:12:15.849 DEBUG:boto:Params: {} 2024-04-09T22:12:15.849 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:12:15.849 DEBUG:boto:Token: None 2024-04-09T22:12:15.850 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:12:15 GMT /xryqqk-36/ 2024-04-09T22:12:15.850 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pRIoUbs0j2YAZy+AsNSWGhGp8L4= 2024-04-09T22:12:15.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pRIoUbs0j2YAZy+AsNSWGhGp8L4=', 'Content-Length': '0'} 2024-04-09T22:12:15.908 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008f0a65482ad77d13-006615bd3f-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:12:15.908 DEBUG:boto:path=/xryqqk-35/ 2024-04-09T22:12:15.908 DEBUG:boto:auth_path=/xryqqk-35/ 2024-04-09T22:12:15.908 DEBUG:boto:Method: HEAD 2024-04-09T22:12:15.908 DEBUG:boto:Path: /xryqqk-35/ 2024-04-09T22:12:15.908 DEBUG:boto:Data: 2024-04-09T22:12:15.908 DEBUG:boto:Headers: {} 2024-04-09T22:12:15.909 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:12:15.909 DEBUG:boto:Port: 8000 2024-04-09T22:12:15.909 DEBUG:boto:Params: {} 2024-04-09T22:12:15.909 DEBUG:boto:Token: None 2024-04-09T22:12:15.909 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:12:15 GMT /xryqqk-35/ 2024-04-09T22:12:15.909 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZjW0F0f0t+FrfHdQpqDqHK3RhqQ= 2024-04-09T22:12:15.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZjW0F0f0t+FrfHdQpqDqHK3RhqQ=', 'Content-Length': '0'} 2024-04-09T22:12:15.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000528a9af1c95bfb37-006615bd3f-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:12:15.912 DEBUG:boto:path=/xryqqk-35/key 2024-04-09T22:12:15.912 DEBUG:boto:auth_path=/xryqqk-35/key 2024-04-09T22:12:15.913 DEBUG:boto:Method: PUT 2024-04-09T22:12:15.913 DEBUG:boto:Path: /xryqqk-35/key 2024-04-09T22:12:15.913 DEBUG:boto:Data: 2024-04-09T22:12:15.913 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-09T22:12:15.913 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:12:15.913 DEBUG:boto:Port: 8000 2024-04-09T22:12:15.913 DEBUG:boto:Params: {} 2024-04-09T22:12:15.913 DEBUG:boto:Token: None 2024-04-09T22:12:15.913 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 09 Apr 2024 22:12:15 GMT /xryqqk-35/key 2024-04-09T22:12:15.913 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:J+72Y9FnecE4ZpOj4dGPbUVmthU= 2024-04-09T22:12:15.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J+72Y9FnecE4ZpOj4dGPbUVmthU='} 2024-04-09T22:12:15.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b79ad6834e8f78a-006615bd3f-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:12:15.919 DEBUG:boto:path=/xryqqk-36/ 2024-04-09T22:12:15.920 DEBUG:boto:auth_path=/xryqqk-36/ 2024-04-09T22:12:15.920 DEBUG:boto:Method: HEAD 2024-04-09T22:12:15.920 DEBUG:boto:Path: /xryqqk-36/ 2024-04-09T22:12:15.920 DEBUG:boto:Data: 2024-04-09T22:12:15.920 DEBUG:boto:Headers: {} 2024-04-09T22:12:15.920 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:12:15.920 DEBUG:boto:Port: 8001 2024-04-09T22:12:15.920 DEBUG:boto:Params: {} 2024-04-09T22:12:15.920 DEBUG:boto:Token: None 2024-04-09T22:12:15.920 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:12:15 GMT /xryqqk-36/ 2024-04-09T22:12:15.921 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JxJ2RtGMEolNEnXlyu8TSswemgs= 2024-04-09T22:12:15.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JxJ2RtGMEolNEnXlyu8TSswemgs=', 'Content-Length': '0'} 2024-04-09T22:12:15.922 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b67345ae39fd3847-006615bd3f-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:12:15.923 DEBUG:boto:path=/xryqqk-36/key 2024-04-09T22:12:15.923 DEBUG:boto:auth_path=/xryqqk-36/key 2024-04-09T22:12:15.923 DEBUG:boto:Method: PUT 2024-04-09T22:12:15.923 DEBUG:boto:Path: /xryqqk-36/key 2024-04-09T22:12:15.923 DEBUG:boto:Data: 2024-04-09T22:12:15.923 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-09T22:12:15.923 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:12:15.923 DEBUG:boto:Port: 8001 2024-04-09T22:12:15.923 DEBUG:boto:Params: {} 2024-04-09T22:12:15.924 DEBUG:boto:Token: None 2024-04-09T22:12:15.924 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Tue, 09 Apr 2024 22:12:15 GMT /xryqqk-36/key 2024-04-09T22:12:15.924 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JHKONE5O7m5J1LxinfBuqHT4lqs= 2024-04-09T22:12:15.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JHKONE5O7m5J1LxinfBuqHT4lqs='} 2024-04-09T22:12:15.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011c5211e82ee8231-006615bd3f-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:12:15.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:15.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:15.931 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:15.979 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:15.979 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:16.008 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700735.871719_444.1", 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:12:15.871719Z" 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:12:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:16.022 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:12:16.022 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:16.023 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:12:16.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:16.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:16.023 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:16.073 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:16.073 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:16.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:16.101+0000 7fe717f83780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:16.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:16.102+0000 7fe717f83780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:16.115 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:16Z" 2024-04-09T22:12:16.121 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:16.122 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:16Z" } 2024-04-09T22:12:16.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:16.122 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:16.122 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:16.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:12:21.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:21.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:21.125 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:21.178 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:21.179 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:21.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:21.208+0000 7f10e7bcf780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:21.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:21.209+0000 7f10e7bcf780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:21.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:21.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:21.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:21.227 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:21.228 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:21Z" 2024-04-09T22:12:21.229 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:21.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:21Z" } 2024-04-09T22:12:21.231 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:21.231 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:21.231 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:21.232 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-09T22:12:26.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:26.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:26.233 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:26.290 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:26.290 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:26.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:26.322+0000 7fe297f42780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:26.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:26.323+0000 7fe297f42780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:26.338 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:26.338 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:26.338 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:26.338 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:26.339 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:26.340 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:26.341 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.892298_376.1", 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:26.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:26.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:26Z" 2024-04-09T22:12:26.344 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:26.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.892298_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.892298Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:26Z" } 2024-04-09T22:12:26.346 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.892298_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.892298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:26.346 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:26.346 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.892298_376.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:26.346 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.892298_376.1 2024-04-09T22:12:31.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:31.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:31.348 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:31.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:31.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:31.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:31.524+0000 7f2be51e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:31.525+0000 7f2be51e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:31.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:31.546 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:31.547 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:31.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:31Z" 2024-04-09T22:12:31.549 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:31.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:31Z" } 2024-04-09T22:12:31.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:31.551 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:31.551 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:31.551 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-09T22:12:36.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:36.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:36.553 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:36.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:36.631+0000 7f6c66484780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:36.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:36.632+0000 7f6c66484780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:36.646 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:36.647 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:36.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:36.649 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:36.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:36.651 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:36Z" 2024-04-09T22:12:36.652 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:36.653 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:36Z" } 2024-04-09T22:12:36.653 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:36.653 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:36.654 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:36.654 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:12:41.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:41.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:41.655 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:41.705 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:41.705 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:41.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:41.734+0000 7f95ab53c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:41.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:41.736+0000 7f95ab53c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:41.749 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:41.750 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:41.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:41.752 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:41.753 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:41.754 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:41.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:41.756 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:41Z" 2024-04-09T22:12:41.756 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:41.757 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:41Z" } 2024-04-09T22:12:41.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:41.757 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:41.757 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:41.758 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.727311_361.1 2024-04-09T22:12:46.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:46.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:46.759 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:46.816 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:46.816 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:46.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:46.848+0000 7f73d5088780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:46.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:46.849+0000 7f73d5088780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:46.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:46.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:46.869 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:46.870 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:46Z" 2024-04-09T22:12:46.871 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:46.871 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:46Z" } 2024-04-09T22:12:46.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:46.872 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:46.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:46.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-09T22:12:51.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:51.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:51.873 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:52.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:52.020+0000 7f3bb87d0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:52.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:52.021+0000 7f3bb87d0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:52.035 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:52.035 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:52.036 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:52.037 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:12:52.038 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:52.039 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:52.040 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:52.041 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:52Z" 2024-04-09T22:12:52.041 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:52.042 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:52Z" } 2024-04-09T22:12:52.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:52.042 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:52.042 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:52.043 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:12:57.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:57.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:12:57.044 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:12:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:12:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:12:57.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:57.124+0000 7f650655e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:57.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:12:57.125+0000 7f650655e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:12:57.138 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:12:57.138 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:57.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:57.140 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:57.141 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:12:57.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:57.143 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:12:57.144 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:12:57.144 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:12:57.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:12:57.144 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:12:57Z" 2024-04-09T22:12:57.144 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:12:57.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:12:57Z" } 2024-04-09T22:12:57.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:12:57.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:57.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:12:57.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-09T22:13:02.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:02.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:02.147 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:02.201 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:02.202 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:02.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:02.233+0000 7fdc39f50780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:02.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:02.234+0000 7fdc39f50780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:02.248 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:02.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:02.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:02.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:02.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:02Z" 2024-04-09T22:13:02.254 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:02.255 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:02Z" } 2024-04-09T22:13:02.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:02.255 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:02.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:02.256 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-09T22:13:07.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:07.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:07.257 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:07.313 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:07.313 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:07.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:07.344+0000 7fae7e183780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:07.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:07.345+0000 7fae7e183780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:07.358 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:07.359 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:07.360 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:07.361 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:07.362 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:07.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:07.364 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:07Z" 2024-04-09T22:13:07.364 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:07.365 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:07Z" } 2024-04-09T22:13:07.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:07.365 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:07.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:07.366 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:13:12.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:12.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:12.367 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:12.444 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:12.444 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:12.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:12.476+0000 7fefddcb5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:12.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:12.477+0000 7fefddcb5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:12.490 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:12.491 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:12.492 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:13:12.493 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:12Z" 2024-04-09T22:13:12.495 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:12.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:12Z" } 2024-04-09T22:13:12.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:12.497 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:12.497 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:12.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:13:17.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:17.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:17.499 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:17.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:17.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:17.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:17.579+0000 7fb83a9ed780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:17.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:17.580+0000 7fb83a9ed780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:17.593 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:17Z" 2024-04-09T22:13:17.599 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:17.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:17Z" } 2024-04-09T22:13:17.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:17.600 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:17.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:17.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-09T22:13:22.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:22.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:22.602 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:22.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:22.657 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:22.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:22.688+0000 7f45619e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:22.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:22.689+0000 7f45619e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:22.703 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:13:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:22.707 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:22.708 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:22.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:22.709 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:22Z" 2024-04-09T22:13:22.709 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:22.710 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:22Z" } 2024-04-09T22:13:22.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:22.710 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:22.710 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:22.710 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:13:27.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:27.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:27.712 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:27.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:27.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:27.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:27.862+0000 7f9475f12780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:27.863+0000 7f9475f12780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:27.878 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:27.878 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:27.878 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:27.878 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:27.878 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:27.881 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:27.881 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:27.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:27.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:27.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:27.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:27.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:27.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.673463_351.1", 2024-04-09T22:13:27.885 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:27.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:27.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:27.886 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:27.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:27.888 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:27.888 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:27.888 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:27.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:27.888 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:27Z" 2024-04-09T22:13:27.888 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:27.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.673463_351.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.673463Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:27Z" } 2024-04-09T22:13:27.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:27.889 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:27.890 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:27.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.673463_351.1 2024-04-09T22:13:32.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:32.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:32.891 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:32.943 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:32.943 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:32.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:32.970+0000 7fdb688d6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:32.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:32.971+0000 7fdb688d6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:32.985 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:32.986 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:32.987 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:32.988 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:32.989 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:32Z" 2024-04-09T22:13:32.990 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:32.991 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:32Z" } 2024-04-09T22:13:32.991 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:32.992 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:32.992 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:32.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-09T22:13:37.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:37.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:37.993 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:38.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:38.080+0000 7ff862b89780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:38.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:38.081+0000 7ff862b89780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:38.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:38.097 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:38.098 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:13:38.099 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:38.100 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:38.101 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:38.102 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:38.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:38.102 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:38Z" 2024-04-09T22:13:38.102 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:38.103 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:38Z" } 2024-04-09T22:13:38.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:38.103 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:38.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:38.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-09T22:13:43.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:43.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:43.105 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:43.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:43.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:43.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:43.184+0000 7f0fa1df3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:43.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:43.185+0000 7f0fa1df3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:43.199 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.694451_356.1", 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.694451Z", 2024-04-09T22:13:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:43Z" 2024-04-09T22:13:43.206 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:43.207 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.694451_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.694451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:43Z" } 2024-04-09T22:13:43.207 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.694451_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.694451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:43.207 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:43.207 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.694451_356.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:43.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.694451_356.1 2024-04-09T22:13:48.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:48.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:48.209 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:48.270 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:48.270 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:48.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:48.300+0000 7efd5ef73780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:48.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:48.301+0000 7efd5ef73780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:48.315 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:48.316 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:48.317 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:48.318 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:48.319 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:48.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:48.321 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:48Z" 2024-04-09T22:13:48.321 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:48.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:48Z" } 2024-04-09T22:13:48.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:48.322 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:48.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:48.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:13:53.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:53.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:53.325 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:53.377 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:53.377 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:53.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:53.407+0000 7fb660878780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:53.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:53.408+0000 7fb660878780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:53.422 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:53.423 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:53.424 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:13:53.425 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:53.426 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:53.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:53.428 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:53Z" 2024-04-09T22:13:53.428 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:53.428 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:53Z" } 2024-04-09T22:13:53.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:53.429 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:53.429 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:53.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-09T22:13:58.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:58.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:13:58.431 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:13:58.483 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:13:58.484 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:13:58.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:58.514+0000 7ff28bf8f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:58.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:13:58.515+0000 7ff28bf8f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:58.529 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:58.530 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:13:58.531 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:13:58.532 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:13:58.533 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:13:58Z" 2024-04-09T22:13:58.534 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:13:58.535 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:13:58Z" } 2024-04-09T22:13:58.535 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:13:58.536 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:58.536 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:13:58.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:14:03.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:03.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:03.537 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:03.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:03.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:03.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:03.700+0000 7f0127cc6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:03.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:03.701+0000 7f0127cc6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:03.716 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:03.716 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:03.717 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:03.718 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:03.719 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.692156_355.1", 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:03.722 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:03.722 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:03.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:03.722 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:03Z" 2024-04-09T22:14:03.722 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:03.723 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.692156_355.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.692156Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:03Z" } 2024-04-09T22:14:03.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:03.723 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:03.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:03.723 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.692156_355.1 2024-04-09T22:14:08.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:08.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:08.724 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:08.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:08.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:08.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:08.802+0000 7fb9140c4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:08.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:08.803+0000 7fb9140c4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:08.817 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:08.817 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:08.817 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:08.817 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:08.817 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:08.817 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:08.818 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:08.819 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:14:08.820 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:08.821 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:08.822 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:08.823 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:08.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:08.823 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:08Z" 2024-04-09T22:14:08.823 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:08.824 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:08Z" } 2024-04-09T22:14:08.824 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:08.824 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:08.824 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:08.824 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:14:13.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:13.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:13.826 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:13.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:13.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:13.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:13.908+0000 7fdb475f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:13.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:13.909+0000 7fdb475f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:13.924 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:13.925 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:13.926 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:13.927 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:13.928 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:13.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:13.930 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:13.930 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:13.930 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:13.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:13.930 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:13Z" 2024-04-09T22:14:13.930 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:13.931 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:13Z" } 2024-04-09T22:14:13.931 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:13.931 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:13.931 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:13.932 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:14:18.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:18.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:18.933 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:18.988 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:18.988 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:19.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:19.020+0000 7f8064bbf780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:19.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:19.021+0000 7f8064bbf780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:19.036 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:19.036 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:19.036 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:19.036 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:19.037 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:19.038 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:19.039 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:19.040 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:19Z" 2024-04-09T22:14:19.041 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:19.042 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:19Z" } 2024-04-09T22:14:19.042 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:19.042 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:19.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:19.043 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:14:24.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:24.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:24.045 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:24.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:24.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:24.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:24.123+0000 7f3eb45b7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:24.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:24.124+0000 7f3eb45b7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:24.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:24.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:24.140 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:24.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:24.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:24.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:24.144 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:24Z" 2024-04-09T22:14:24.144 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:24.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:24Z" } 2024-04-09T22:14:24.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:24.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:24.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:24.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:14:29.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:29.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:29.147 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:29.199 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:29.199 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:29.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:29.227+0000 7fc2b0cff780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:29.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:29.228+0000 7fc2b0cff780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:29.242 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:29.242 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:29.242 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:29.242 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:29.242 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:29.242 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:29.243 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:29.244 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:29.245 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:29.246 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:29.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:29.248 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:29Z" 2024-04-09T22:14:29.248 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:29.249 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:29Z" } 2024-04-09T22:14:29.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:29.249 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:29.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:29.249 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.727311_361.1 2024-04-09T22:14:34.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:34.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:34.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:34.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:34.335+0000 7fa3becf9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:34.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:34.336+0000 7fa3becf9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:34.351 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:34.355 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:34.356 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:34.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:34.357 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:34.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:34.357 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:34Z" 2024-04-09T22:14:34.357 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:34.358 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:34Z" } 2024-04-09T22:14:34.358 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:34.358 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:34.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:34.358 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:14:39.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:39.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:39.359 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:39.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:39.468+0000 7f7f3c0ac780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:39.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:39.471+0000 7f7f3c0ac780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:39.486 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:39.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:39.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:39.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:39.490 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:39Z" 2024-04-09T22:14:39.491 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:39.492 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:39Z" } 2024-04-09T22:14:39.492 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:39.492 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:39.493 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:39.493 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:14:44.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:44.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:44.494 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:44.548 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:44.548 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:44.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:44.580+0000 7f81732f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:44.581+0000 7f81732f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:44.596 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:44.597 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:44.598 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:44.599 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:44.600 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:44Z" 2024-04-09T22:14:44.601 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:44.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:44Z" } 2024-04-09T22:14:44.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:44.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:44.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:44.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.813227_371.1 2024-04-09T22:14:49.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:49.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:49.604 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:49.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:49.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:49.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:49.686+0000 7fe899004780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:49.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:49.687+0000 7fe899004780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:49.701 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:49.702 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:49.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:49.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:49.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:49.706 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:49.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:49.707 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:49Z" 2024-04-09T22:14:49.707 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:49.708 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:49Z" } 2024-04-09T22:14:49.708 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:49.708 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:49.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:49.709 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-09T22:14:54.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:54.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:54.711 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:54.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:54.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:14:54.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:54.810+0000 7f6fe4a27780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:54.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:14:54.811+0000 7f6fe4a27780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:54.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:54.828 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:54.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:54.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:14:54.831 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:14:54Z" 2024-04-09T22:14:54.832 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:14:54.833 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:14:54Z" } 2024-04-09T22:14:54.833 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:14:54.833 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:54.833 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:14:54.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-09T22:14:59.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:59.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:14:59.835 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:14:59.937 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:14:59.937 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:00.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:00.009+0000 7fa63ca5c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:00.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:00.010+0000 7fa63ca5c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:00.024 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:00.025 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:00.026 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.694451_356.1", 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:00.027 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.694451Z", 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:00.028 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:00.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:00.030 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:00Z" 2024-04-09T22:15:00.030 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:00.031 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.694451_356.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.694451Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:00Z" } 2024-04-09T22:15:00.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.694451_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.694451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:00.031 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:00.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.694451_356.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:00.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.694451_356.1 2024-04-09T22:15:05.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:05.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:05.033 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:05.084 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:05.085 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:05.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:05.119+0000 7fd713d9a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:05.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:05.120+0000 7fd713d9a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:05.137 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:05.137 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:05.138 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:05.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:05.140 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:05.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:05.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:05.143 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:05.144 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:05Z" 2024-04-09T22:15:05.144 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:05.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:05Z" } 2024-04-09T22:15:05.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:05.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:05.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:05.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:15:10.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:10.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:10.147 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:10.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:10.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:10.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:10.239+0000 7f9f352b8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:10.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:10.240+0000 7f9f352b8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:10.256 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:10.256 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:10.256 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:10.257 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:10.257 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:10.257 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:10.257 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:10.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:10.258 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:10.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:10.258 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:10.258 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:10.258 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:10.259 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:10.259 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:10.259 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:10.259 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:10.259 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:10.260 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:10.261 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:10.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:10.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:10Z" 2024-04-09T22:15:10.264 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:10.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:10Z" } 2024-04-09T22:15:10.266 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:10.266 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:10.266 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:10.266 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-09T22:15:15.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:15.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:15.268 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:15.343 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:15.343 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:15.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:15.460+0000 7f7b995e0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:15.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:15.461+0000 7f7b995e0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:15.476 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:15Z" 2024-04-09T22:15:15.482 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:15.483 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:15Z" } 2024-04-09T22:15:15.484 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:15.484 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:15.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:15.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.798600_369.1 2024-04-09T22:15:20.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:20.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:20.485 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:20.537 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:20.538 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:20.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:20.566+0000 7fa731c36780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:20.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:20.567+0000 7fa731c36780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:20.581 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:20.581 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:20.581 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:20.581 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:20.581 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:20.582 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:20.583 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.756367_365.1", 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:20.584 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:20.585 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:20.586 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:20.587 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:20.587 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:20.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:20.587 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:20Z" 2024-04-09T22:15:20.587 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:20.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.756367_365.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.756367Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:20Z" } 2024-04-09T22:15:20.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:20.588 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:20.588 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:20.588 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.756367_365.1 2024-04-09T22:15:25.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:25.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:25.590 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:25.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:25.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:25.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:25.686+0000 7f774fb71780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:25.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:25.687+0000 7f774fb71780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:25.701 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:25.701 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:25.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:25.704 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:25.705 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:25.706 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:25.707 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:25.707 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:25.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:25.707 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:25Z" 2024-04-09T22:15:25.707 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:25.708 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:25Z" } 2024-04-09T22:15:25.708 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:25.708 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:25.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:25.708 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:15:30.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:30.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:30.710 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:30.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:30.768 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:30.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:30.801+0000 7f8777de7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:30.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:30.803+0000 7f8777de7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:30.819 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:30.819 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:30.819 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:30.819 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:30.819 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:30.819 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:30.820 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:30.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:30.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:30.822 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:30.822 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:30.822 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:30.822 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:30.823 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:15:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:30Z" 2024-04-09T22:15:30.828 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:30.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:30Z" } 2024-04-09T22:15:30.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:30.829 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:30.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:30.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.704141_357.1 2024-04-09T22:15:35.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:35.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:35.831 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:35.881 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:35.882 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:35.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:35.923+0000 7f788c7ff780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:35.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:35.924+0000 7f788c7ff780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:35.938 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:35.939 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:35.940 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:35.941 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:35.942 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:35.942 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:35.942 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:35.942 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:35.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:35.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.675822_352.1", 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:35.943 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:35.944 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:35.945 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:35.946 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:35Z" 2024-04-09T22:15:35.947 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:35.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.675822_352.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.675822Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:35Z" } 2024-04-09T22:15:35.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:35.948 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:35.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:35.948 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.675822_352.1 2024-04-09T22:15:40.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:40.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:40.950 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:41.001 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:41.001 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:41.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:41.039+0000 7f3f100bb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:41.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:41.040+0000 7f3f100bb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:41.059 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:41.059 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:41.059 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:41.059 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:41.060 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:41.061 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:41.062 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:41.063 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:41.063 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:41.063 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:15:41.063 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:41.063 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:41.063 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:41.064 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:41.065 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:41.065 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:41.065 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:41.065 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:41.065 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:41.066 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:41.066 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:41.066 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:41.067 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:41.067 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:41.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:41.067 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:41.067 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:41.068 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:41.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:41.068 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:41Z" 2024-04-09T22:15:41.068 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:41.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:41Z" } 2024-04-09T22:15:41.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:41.070 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:41.071 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:41.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-09T22:15:46.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:46.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:46.073 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:46.125 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:46.125 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:46.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:46.169+0000 7fa14a3ec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:46.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:46.170+0000 7fa14a3ec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:46.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:46.199 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:46Z" 2024-04-09T22:15:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:46.201 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:46Z" } 2024-04-09T22:15:46.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:46.202 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:46.202 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:46.202 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:15:51.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:51.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:51.204 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:51.261 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:51.261 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:51.298+0000 7f49fd94d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:51.299+0000 7f49fd94d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:51.315 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:51.315 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:51.315 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:51.315 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:51.316 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:51.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:51.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:51.319 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:51.320 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:51Z" 2024-04-09T22:15:51.321 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:51.322 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:51Z" } 2024-04-09T22:15:51.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:51.323 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:51.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:51.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-09T22:15:56.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:56.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:15:56.325 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:15:56.414 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:15:56.415 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:15:56.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:56.482+0000 7f1e32530780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:56.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:15:56.483+0000 7f1e32530780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:15:56.498 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:56.499 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:56.500 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:56.501 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:15:56.502 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:15:56Z" 2024-04-09T22:15:56.503 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:15:56.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:15:56Z" } 2024-04-09T22:15:56.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:15:56.505 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:56.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:15:56.505 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.813227_371.1 2024-04-09T22:16:01.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:01.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:01.506 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:01.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:01.586+0000 7f15d0f03780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:01.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:01.587+0000 7f15d0f03780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:01.605 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:01.605 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:01.605 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:01.605 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:01.605 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:01.605 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:01.606 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:01.607 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:01.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:01.609 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:01.610 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:01.611 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:01Z" 2024-04-09T22:16:01.612 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:01.612 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:01Z" } 2024-04-09T22:16:01.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:01.612 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:01.612 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:01.613 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.704141_357.1 2024-04-09T22:16:06.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:06.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:06.614 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:06.671 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:06.671 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:06.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:06.709+0000 7fef4c2b8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:06.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:06.710+0000 7fef4c2b8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:06.726 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:06.730 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:06.731 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:06.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:06.733 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:06.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:06Z" 2024-04-09T22:16:06.736 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:06.737 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:06Z" } 2024-04-09T22:16:06.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:06.738 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:06.739 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:06.739 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:16:11.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:11.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:11.741 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:11.812 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:11.813 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:11.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:11.949+0000 7f5819875780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:11.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:11.950+0000 7f5819875780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:11.977 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:11.977 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:11.977 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:11.977 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:11.978 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:11.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:11.980 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:11.981 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.675822_352.1", 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:11.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:11.983 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:11.984 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:11.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:11.984 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:11.984 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:11.985 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:11.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:11.985 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:11Z" 2024-04-09T22:16:11.985 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:11.986 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.675822_352.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.675822Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:11Z" } 2024-04-09T22:16:11.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:11.986 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:11.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:11.987 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.675822_352.1 2024-04-09T22:16:16.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:16.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:16.988 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:17.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:17.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:17.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:17.072+0000 7f974861c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:17.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:17.073+0000 7f974861c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:17.088 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:17Z" 2024-04-09T22:16:17.095 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:17.095 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:17Z" } 2024-04-09T22:16:17.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:17.096 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:17.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:17.096 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:16:22.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:22.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:22.097 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:22.150 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:22.151 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:22.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:22.200+0000 7f33198f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:22.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:22.201+0000 7f33198f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:22.216 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:22.216 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:22.217 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:22.218 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:22.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:22.220 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:22.220 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:22.220 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:22.220 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:16:22.220 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:22.220 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:22.221 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:22.222 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:22.222 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:22.222 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:22.222 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:22.222 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:22.223 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:22Z" 2024-04-09T22:16:22.224 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:22.225 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:22Z" } 2024-04-09T22:16:22.225 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:22.225 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:22.225 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:22.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.813227_371.1 2024-04-09T22:16:27.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:27.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:27.226 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:27.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:27.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:27.334+0000 7f2e63a2e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:27.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:27.335+0000 7f2e63a2e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:27.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:27.359 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:27.360 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.663981_350.1", 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:27.361 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:27.362 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:27Z" 2024-04-09T22:16:27.363 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:27.365 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.663981_350.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.663981Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:27Z" } 2024-04-09T22:16:27.365 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:27.365 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:27.365 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:27.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-09T22:16:32.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:32.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:32.367 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:32.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:32.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:32.474+0000 7f512a860780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:32.476+0000 7f512a860780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:32.493 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:32.493 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:32.493 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:32.493 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:32.493 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:32.493 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:32.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:32.495 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:32.496 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:32.497 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:32.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:32.499 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:32Z" 2024-04-09T22:16:32.499 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:32.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:32Z" } 2024-04-09T22:16:32.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:32.500 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:32.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:32.500 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:16:37.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:37.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:37.502 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:37.552 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:37.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:37.599+0000 7f6bbc02b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:37.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:37.600+0000 7f6bbc02b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:37.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:37.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:37.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.642202_346.1", 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:37.626 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:37.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:37Z" 2024-04-09T22:16:37.628 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:37.629 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.642202_346.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.642202Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:37Z" } 2024-04-09T22:16:37.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:37.629 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:37.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:37.630 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-09T22:16:42.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:42.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:42.632 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:42.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:42.683 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:42.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:42.728+0000 7fc924064780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:42.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:42.729+0000 7fc924064780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:42.743 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:42.743 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:42.744 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:42.745 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:42.746 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.661401_349.1", 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:42.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:42.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:42Z" 2024-04-09T22:16:42.749 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:42.750 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.661401_349.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.661401Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:42Z" } 2024-04-09T22:16:42.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:42.750 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:42.750 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:42.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-09T22:16:47.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:47.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:47.752 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:47.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:47.846+0000 7f9a51973780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:47.847+0000 7f9a51973780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:47.862 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:47.862 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:47.862 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:47.862 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:47.862 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:47.862 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:47.863 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:47.864 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:47.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.675822_352.1", 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:47.866 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:47.867 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:47Z" 2024-04-09T22:16:47.868 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:47.869 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.675822_352.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.675822Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:47Z" } 2024-04-09T22:16:47.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:47.870 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:47.870 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:47.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.675822_352.1 2024-04-09T22:16:52.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:52.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:52.871 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:52.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:52.973+0000 7f6af6816780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:52.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:52.974+0000 7f6af6816780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:52.991 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:52.991 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:52.991 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:52.991 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:52.992 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:52.993 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:52.994 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.704141_357.1", 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:52.995 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:52.996 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:52.997 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:52.997 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:52.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:52.997 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:52Z" 2024-04-09T22:16:52.997 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:52.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.704141_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.704141Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:52Z" } 2024-04-09T22:16:52.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:52.999 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:52.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:52.999 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.704141_357.1 2024-04-09T22:16:58.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:58.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:16:58.001 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:16:58.104 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:16:58.104 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:16:58.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:58.179+0000 7f0d1ee71780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:58.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:16:58.180+0000 7f0d1ee71780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:16:58.216 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:16:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:16:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:16:58.216 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:16:58.217 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:16:58.218 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:58.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.651742_348.1", 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-09T22:16:58.220 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:16:58.221 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:16:58Z" 2024-04-09T22:16:58.222 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:16:58.223 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.651742_348.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.651742Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:16:58Z" } 2024-04-09T22:16:58.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:16:58.223 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:58.224 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:16:58.224 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-09T22:17:03.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:03.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:03.225 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:17:03.276 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:17:03.276 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:17:03.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:03.311+0000 7f6a89d1d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:03.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:03.312+0000 7f6a89d1d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:03.327 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:17:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:17:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:17:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:17:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:17:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:17:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:17:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:17:03.333 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:03.333 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:03.333 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:17:03.333 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:03.333 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:03.334 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:03.334 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:17:03.334 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:03.334 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:03.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:03.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:17:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:17:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:17:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:17:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:17:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:17:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:17:03Z" 2024-04-09T22:17:03.338 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:17:03.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:17:03Z" } 2024-04-09T22:17:03.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:17:03.340 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:03.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:03.340 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:17:08.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:08.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:08.341 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:17:08.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:17:08.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:17:08.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:08.448+0000 7f23a92c3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:08.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:08.449+0000 7f23a92c3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:17:08.472 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:08.473 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:08.474 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:08.475 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.649397_347.1", 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:17:08Z" 2024-04-09T22:17:08.478 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:17:08.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.649397_347.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.649397Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:17:08Z" } 2024-04-09T22:17:08.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:17:08.479 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:08.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:08.480 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-09T22:17:13.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:13.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:13.481 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:17:13.539 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:17:13.539 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:17:13.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:13.579+0000 7fca5772d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:13.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:13.580+0000 7fca5772d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:13.596 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:17:13.597 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:13.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:13.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.688829_353.1", 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:13.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:13.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:17:13Z" 2024-04-09T22:17:13.602 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:17:13.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.688829_353.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.688829Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:17:13Z" } 2024-04-09T22:17:13.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:17:13.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:13.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:13.604 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-09T22:17:18.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:18.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:17:18.605 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:17:18.658 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:17:18.658 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:17:18.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:18.711+0000 7ffa8a505780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:18.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:17:18.712+0000 7ffa8a505780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:17:18.727 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:17:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:17:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:17:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:17:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:17:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.673463_351.1", 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:18.733 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:17:18.733 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:17:18.733 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:17:18.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:18.733 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:17:18Z" 2024-04-09T22:17:18.733 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:17:18.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.673463_351.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.673463Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:17:18Z" } 2024-04-09T22:17:18.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:17:18.734 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:18.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:17:18.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.673463_351.1 2024-04-09T22:17:23.740 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-04-09T22:17:23.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:17:23.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:17:23.742 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:17:23.790 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:17:23.790 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "a2.rgw.control", 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-09T22:17:23.811 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "a2.rgw.log", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "a2.rgw.otp", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T22:17:23.812 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: "compression_type": "random" 2024-04-09T22:17:23.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-09T22:17:23.814 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:17:23.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:17:23.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:17:23.816 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:17:23.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:17:23.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:17:23.930 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone a5723776-f142-428c-a6df-d7bf949257da 2024-04-09T22:17:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 5, 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T22:17:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:17:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:17:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:17:23.975 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:17:23.976 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:17:23.977 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:17:23.979 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:17:23.980 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:17:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:17:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:17:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:17:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:17:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:17:23.981 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:17:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:17:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.987 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:17:23.988 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.989 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:17:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.990 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.991 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T22:17:23.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T22:17:23.993 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:17:23.994 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:17:23.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:17:23.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:17:23.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:17:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:17:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T22:17:23.998 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:17:23.998 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:17:24.002 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2024-04-09T22:18:24.003 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-37 2024-04-09T22:18:24.004 DEBUG:boto:path=/xryqqk-37/ 2024-04-09T22:18:24.004 DEBUG:boto:auth_path=/xryqqk-37/ 2024-04-09T22:18:24.004 DEBUG:boto:Method: PUT 2024-04-09T22:18:24.004 DEBUG:boto:Path: /xryqqk-37/ 2024-04-09T22:18:24.004 DEBUG:boto:Data: 2024-04-09T22:18:24.005 DEBUG:boto:Headers: {} 2024-04-09T22:18:24.005 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:18:24.005 DEBUG:boto:Port: 8000 2024-04-09T22:18:24.005 DEBUG:boto:Params: {} 2024-04-09T22:18:24.005 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:18:24.006 DEBUG:boto:Token: None 2024-04-09T22:18:24.006 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:18:24 GMT /xryqqk-37/ 2024-04-09T22:18:24.006 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WmTusarOm4DJfzSD/pmFQtLa15k= 2024-04-09T22:18:24.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:18:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WmTusarOm4DJfzSD/pmFQtLa15k=', 'Content-Length': '0'} 2024-04-09T22:18:24.031 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000150c08023d93ef76-006615beb0-1842-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:18:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:18:24.032 DEBUG:boto:path=/xryqqk-37/testredirect 2024-04-09T22:18:24.032 DEBUG:boto:auth_path=/xryqqk-37/testredirect 2024-04-09T22:18:24.032 DEBUG:boto:Method: PUT 2024-04-09T22:18:24.032 DEBUG:boto:Path: /xryqqk-37/testredirect 2024-04-09T22:18:24.032 DEBUG:boto:Data: 2024-04-09T22:18:24.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-09T22:18:24.032 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:18:24.032 DEBUG:boto:Port: 8000 2024-04-09T22:18:24.032 DEBUG:boto:Params: {} 2024-04-09T22:18:24.032 DEBUG:boto:Token: None 2024-04-09T22:18:24.033 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 09 Apr 2024 22:18:24 GMT /xryqqk-37/testredirect 2024-04-09T22:18:24.033 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DKZ0Jpo0BLzz4GdQAGPSlP6LYlQ= 2024-04-09T22:18:24.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:18:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DKZ0Jpo0BLzz4GdQAGPSlP6LYlQ='} 2024-04-09T22:18:24.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dd0ec11c73235e8-006615beb0-1842-a1'), ('Date', 'Tue, 09 Apr 2024 22:18:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:18:24.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:24.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:24.045 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:24.120 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:24.121 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:24.201 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:24.219 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:18:24.029085Z" 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:24.220 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:18:24.221 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:24.222 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:18:24.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:24.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:24.222 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:24.273 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:24.273 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:24.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:24.331+0000 7f1df689a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:24.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:24.332+0000 7f1df689a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:24.352 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:24.353 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:24.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.729688_362.1", 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:24.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:24.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:24Z" 2024-04-09T22:18:24.358 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:24.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.729688_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.729688Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:24Z" } 2024-04-09T22:18:24.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.729688_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.729688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:24.360 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:24.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.729688_362.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:24.360 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.729688_362.1 2024-04-09T22:18:29.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:29.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:29.361 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:29.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:29.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:29.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:29.445+0000 7fa2841da780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:29.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:29.446+0000 7fa2841da780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:29.460 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:29.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:29.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:29.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.756367_365.1", 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:29.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:29.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:29Z" 2024-04-09T22:18:29.466 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:29.467 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.756367_365.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.756367Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:29Z" } 2024-04-09T22:18:29.467 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:29.467 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:29.467 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:29.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.756367_365.1 2024-04-09T22:18:34.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:34.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:34.470 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:34.523 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:34.523 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:34.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:34.558+0000 7fc3fc5fb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:34.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:34.559+0000 7fc3fc5fb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:34.574 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:34.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:34.576 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.773250_367.1", 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:34.577 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:34.578 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:34.579 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:34.580 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:34Z" 2024-04-09T22:18:34.580 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:34.581 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.773250_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.773250Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:34Z" } 2024-04-09T22:18:34.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.773250_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.773250Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:34.581 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:34.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.773250_367.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:34.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.773250_367.1 2024-04-09T22:18:39.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:39.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:39.582 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:39.643 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:39.643 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:39.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:39.677+0000 7f0e985a7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:39.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:39.678+0000 7f0e985a7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:39.693 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:39.693 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:39.693 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:39.693 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:39.693 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:39.693 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:39.694 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:39.695 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.730558_363.1", 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:39.696 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:39.697 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:39.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:39.699 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:39Z" 2024-04-09T22:18:39.699 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:39.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.730558_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.730558Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:39Z" } 2024-04-09T22:18:39.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:39.700 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:39.700 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:39.700 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.730558_363.1 2024-04-09T22:18:44.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:44.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:44.701 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:44.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:44.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:44.799+0000 7f6b98e93780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:44.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:44.800+0000 7f6b98e93780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:44.815 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:44.816 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:44.817 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.889732_375.1", 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:44.819 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:44Z" 2024-04-09T22:18:44.820 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:44.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.889732_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.889732Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:44Z" } 2024-04-09T22:18:44.822 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:44.822 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:44.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:44.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-09T22:18:49.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:49.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:49.823 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:49.880 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:49.880 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:49.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:49.921+0000 7fe279640780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:49.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:49.922+0000 7fe279640780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:49.937 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:49.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:49.939 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:49.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700425.006471_386.1", 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:05.006471Z", 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:49.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:49.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:49Z" 2024-04-09T22:18:49.943 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:49.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700425.006471_386.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:05.006471Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:49Z" } 2024-04-09T22:18:49.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700425.006471_386.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:05.006471Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:49.944 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:49.944 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700425.006471_386.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:49.945 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700425.006471_386.1 2024-04-09T22:18:54.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:54.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:18:54.946 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:18:54.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:18:54.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:18:55.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:55.031+0000 7f2198f29780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:55.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:18:55.032+0000 7f2198f29780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:18:55.046 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:18:55.046 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:18:55.046 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:18:55.046 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:18:55.046 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:18:55.047 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:18:55.047 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:55.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:55.047 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:18:55.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:18:55.048 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:55.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:55.050 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:18:55.051 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:18:55.052 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:18:55.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:18:55.053 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:18:55Z" 2024-04-09T22:18:55.053 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:18:55.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:18:55Z" } 2024-04-09T22:18:55.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:18:55.055 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:55.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:18:55.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:19:00.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:00.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:00.057 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:00.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:00.155+0000 7faadc564780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:00.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:00.156+0000 7faadc564780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:00.179 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:00.179 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:00.179 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:00.179 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:00.180 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:00.181 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:00.182 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.949651_378.1", 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.949651Z", 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:00.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:00.184 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:00.185 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:00.185 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:00.185 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:00.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:00.185 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:00.185 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:00.186 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:00.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:00.187 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:00Z" 2024-04-09T22:19:00.187 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:00.188 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.949651_378.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.949651Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:00Z" } 2024-04-09T22:19:00.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.949651_378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.949651Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:00.188 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:00.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.949651_378.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:00.188 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.949651_378.1 2024-04-09T22:19:05.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:05.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:05.190 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:05.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:05.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:05.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:05.296+0000 7f596792a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:05.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:05.297+0000 7f596792a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:05.314 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:05.315 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:05.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:05.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.730558_363.1", 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:05.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:05.319 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:05Z" 2024-04-09T22:19:05.320 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:05.321 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.730558_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.730558Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:05Z" } 2024-04-09T22:19:05.322 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:05.322 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:05.322 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:05.322 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.730558_363.1 2024-04-09T22:19:10.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:10.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:10.323 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:10.374 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:10.375 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:10.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:10.423+0000 7f0c7f27b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:10.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:10.424+0000 7f0c7f27b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:10.438 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:10.438 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:10.438 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:10.438 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:10.439 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:10.440 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:10.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:10.442 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:10.443 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:10Z" 2024-04-09T22:19:10.444 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:10.446 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:10Z" } 2024-04-09T22:19:10.446 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:10.446 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:10.446 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:10.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:19:15.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:15.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:15.448 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:15.500 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:15.500 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:15.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:15.553+0000 7fbefcd9c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:15.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:15.554+0000 7fbefcd9c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.729688_362.1", 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:15Z" 2024-04-09T22:19:15.580 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:15.580 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.729688_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.729688Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:15Z" } 2024-04-09T22:19:15.581 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.729688_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.729688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:15.581 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:15.581 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.729688_362.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:15.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.729688_362.1 2024-04-09T22:19:20.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:20.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:20.583 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:20.642 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:20.643 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:20.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:20.686+0000 7f920df4b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:20.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:20.687+0000 7f920df4b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:20.708 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:20.708 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:20.708 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:20.708 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:20.708 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:20.708 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:20.709 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:20.710 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:20.711 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:19:20.712 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:20.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:20.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:20.715 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:20Z" 2024-04-09T22:19:20.715 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:20.719 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:20Z" } 2024-04-09T22:19:20.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:20.719 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:20.719 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:20.719 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:19:25.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:25.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:25.721 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:25.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:25.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:25.810+0000 7f6ab332f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:25.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:25.811+0000 7f6ab332f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:25.825 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:25.826 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:25.826 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:25.826 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:25.826 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:25.826 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:25.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:25.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:25.827 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:25.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:25.827 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:25.828 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:25.828 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:25.828 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:25.828 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:25.829 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:25.829 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:25.829 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:25.829 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:25.829 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:25.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:25.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:25.830 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:25.830 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:25.830 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:25.831 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:25.831 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:25.831 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:25.831 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:25.831 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:25.832 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:25.832 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:25.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:25.832 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:25.832 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:25.833 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:25.833 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:25.833 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.730558_363.1", 2024-04-09T22:19:25.833 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:25.833 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:25.834 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:25.834 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-09T22:19:25.834 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:25.834 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:25.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:25.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:25.835 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:25.835 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:25.835 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:25.836 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:25.836 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:25.836 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:25.836 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:25.836 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:25.837 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:25.837 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:25.837 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:25.837 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:25.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:25.838 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:25.838 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:25.838 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:25.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:25.838 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:25Z" 2024-04-09T22:19:25.838 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:25.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.730558_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.730558Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:25Z" } 2024-04-09T22:19:25.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:25.841 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:25.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:25.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.730558_363.1 2024-04-09T22:19:30.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:30.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:30.843 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:30.895 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:30.896 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:30.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:30.943+0000 7f2c5d97a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:30.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:30.944+0000 7f2c5d97a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:30.964 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:30.964 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:30.964 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:30.965 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:30.966 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.709706_360.1", 2024-04-09T22:19:30.967 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:30.968 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:30.969 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:30.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:30.970 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:30Z" 2024-04-09T22:19:30.970 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:30.971 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.709706_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.709706Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:30Z" } 2024-04-09T22:19:30.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:30.971 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:30.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:30.971 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-09T22:19:35.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:35.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:35.973 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:36.030 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:36.031 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:36.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:36.082+0000 7faa8a8df780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:36.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:36.083+0000 7faa8a8df780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:36.100 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:36.100 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:36.100 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:36.100 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:36.100 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:36.100 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:36.101 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:36.102 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.709706_360.1", 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:36.103 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:36.104 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:36Z" 2024-04-09T22:19:36.105 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:36.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.709706_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.709706Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:36Z" } 2024-04-09T22:19:36.107 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:36.107 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:36.107 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:36.107 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-09T22:19:41.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:41.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:41.108 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:41.159 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:41.159 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:41.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:41.210+0000 7f8c7a590780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:41.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:41.211+0000 7f8c7a590780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:41.233 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:41.234 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:41.235 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.729688_362.1", 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:41.236 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:41.237 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:41.238 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:41.238 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:41.238 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:41.239 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:41.239 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:41.239 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:41.239 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:41Z" 2024-04-09T22:19:41.240 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:41.241 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.729688_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.729688Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:41Z" } 2024-04-09T22:19:41.241 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.729688_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.729688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:41.241 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:41.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.729688_362.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:41.242 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.729688_362.1 2024-04-09T22:19:46.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:46.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:46.243 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:46.301 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:46.302 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:46.355+0000 7f94d976f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:46.356+0000 7f94d976f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:46.374 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:46.374 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:46.375 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:46.376 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:46.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:46.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:46.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:46Z" 2024-04-09T22:19:46.380 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:46.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:46Z" } 2024-04-09T22:19:46.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:46.382 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:46.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:46.382 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:19:51.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:51.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:51.384 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:51.435 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:51.435 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:51.486+0000 7fb9d35a7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:51.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:51.487+0000 7fb9d35a7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:51.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:51.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:51.504 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:51.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:51.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:51Z" 2024-04-09T22:19:51.507 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:51.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:51Z" } 2024-04-09T22:19:51.509 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:51.509 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:51.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:51.509 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:19:56.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:56.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:19:56.514 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:19:56.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:19:56.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:19:56.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:56.628+0000 7f008b622780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:56.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:19:56.629+0000 7f008b622780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:56.651 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:56.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:56.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:19:56.654 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:19:56.655 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:19:56Z" 2024-04-09T22:19:56.656 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:19:56.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:19:56Z" } 2024-04-09T22:19:56.657 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:19:56.658 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:56.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:19:56.658 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:20:01.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:01.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:01.660 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:01.753 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:01.753 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:01.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:01.801+0000 7f91b23f1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:01.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:01.802+0000 7f91b23f1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:01.822 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:01.823 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:01.824 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.709706_360.1", 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-09T22:20:01.825 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:01.826 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:01.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:01.828 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:01Z" 2024-04-09T22:20:01.828 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:01.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.709706_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.709706Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:01Z" } 2024-04-09T22:20:01.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:01.829 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:01.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:01.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-09T22:20:06.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:06.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:06.831 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:06.888 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:06.888 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:06.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:06.932+0000 7f2d423ca780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:06.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:06.933+0000 7f2d423ca780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:06.947 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:06.947 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:06.947 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:06.948 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:06.949 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:06.950 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:06.953 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:06.953 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:06.953 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:06.954 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:06.955 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:06.956 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:06.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:06.956 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:06Z" 2024-04-09T22:20:06.956 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:06.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:06Z" } 2024-04-09T22:20:06.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:06.957 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:06.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:06.958 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:20:11.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:11.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:11.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:12.010 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:12.010 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:12.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:12.051+0000 7fa6a6231780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:12.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:12.052+0000 7fa6a6231780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:12.071 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:12.072 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:12.073 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:12.074 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:12.075 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:12Z" 2024-04-09T22:20:12.076 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:12.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:12Z" } 2024-04-09T22:20:12.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:12.078 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:12.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:12.078 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:20:17.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:17.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:17.079 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:17.131 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:17.132 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:17.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:17.172+0000 7f6c9ed3b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:17.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:17.173+0000 7f6c9ed3b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:17Z" 2024-04-09T22:20:17.193 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:17.194 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:17Z" } 2024-04-09T22:20:17.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:17.195 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:17.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:17.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:20:22.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:22.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:22.197 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:22.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:22.249 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:22.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:22.299+0000 7fb6ca9f6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:22.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:22.300+0000 7fb6ca9f6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:22.318 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:22.319 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:22.320 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.946918_377.1", 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:22.321 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:22Z" 2024-04-09T22:20:22.324 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:22.325 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.946918_377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.946918Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:22Z" } 2024-04-09T22:20:22.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.946918_377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.946918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:22.325 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:22.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.946918_377.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:22.326 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.946918_377.1 2024-04-09T22:20:27.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:27.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:27.327 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:27.379 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:27.379 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:27.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:27.423+0000 7f74d444d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:27.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:27.424+0000 7f74d444d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:27.442 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:27.442 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:27.443 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:27.444 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:27.445 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:27.446 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:27.447 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:27Z" 2024-04-09T22:20:27.448 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:27.449 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:27Z" } 2024-04-09T22:20:27.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:27.450 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:27.450 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:27.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:20:32.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:32.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:32.451 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:32.505 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:32.505 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:32.553+0000 7f256d707780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:32.554+0000 7f256d707780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:32.572 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:32.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:32.574 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.946918_377.1", 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:32.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:32.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:32.577 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:32Z" 2024-04-09T22:20:32.578 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:32.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.946918_377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.946918Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:32Z" } 2024-04-09T22:20:32.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.946918_377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.946918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:32.579 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:32.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.946918_377.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:32.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.946918_377.1 2024-04-09T22:20:37.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:37.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:37.581 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:37.632 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:37.632 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:37.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:37.675+0000 7f4789c7d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:37.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:37.676+0000 7f4789c7d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:37.696 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:37.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:37.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.709706_360.1", 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-09T22:20:37.700 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:37.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:37.702 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:37.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:37.703 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:37Z" 2024-04-09T22:20:37.703 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:37.704 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.709706_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.709706Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:37Z" } 2024-04-09T22:20:37.704 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:37.704 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:37.705 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:37.705 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-09T22:20:42.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:42.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:42.707 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:42.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:42.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:42.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:42.805+0000 7f8706e74780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:42.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:42.806+0000 7f8706e74780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:42.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:42.823 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:42.824 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.709706_360.1", 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-09T22:20:42.825 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:42.826 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:42Z" 2024-04-09T22:20:42.827 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:42.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.709706_360.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.709706Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:42Z" } 2024-04-09T22:20:42.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:42.829 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:42.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:42.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-09T22:20:47.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:47.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:47.831 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:47.882 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:47.882 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:47.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:47.916+0000 7ffa909a7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:47.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:47.917+0000 7ffa909a7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.773250_367.1", 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:47.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:47Z" 2024-04-09T22:20:47.939 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:47.940 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.773250_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.773250Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:47Z" } 2024-04-09T22:20:47.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.773250_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.773250Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:47.941 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:47.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.773250_367.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:47.941 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.773250_367.1 2024-04-09T22:20:52.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:52.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:52.942 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:52.998 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:52.999 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:53.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:53.041+0000 7fbcaec83780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:53.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:53.042+0000 7fbcaec83780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:53.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:53.060 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:53.061 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700426.099551_395.1", 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:06.099551Z", 2024-04-09T22:20:53.062 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:53.063 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:53Z" 2024-04-09T22:20:53.064 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:53.065 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700426.099551_395.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:06.099551Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:53Z" } 2024-04-09T22:20:53.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700426.099551_395.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:06.099551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:53.066 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:53.066 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700426.099551_395.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:53.066 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700426.099551_395.1 2024-04-09T22:20:58.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:58.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:20:58.067 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:20:58.126 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:20:58.127 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:20:58.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:58.179+0000 7fe271307780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:58.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:20:58.180+0000 7fe271307780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:20:58.196 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:20:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:20:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:20:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:20:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:20:58.196 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:58.197 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:58.198 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.727311_361.1", 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:58.199 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:20:58.200 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:20:58.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:20:58.202 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:20:58Z" 2024-04-09T22:20:58.202 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:20:58.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.727311_361.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.727311Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:20:58Z" } 2024-04-09T22:20:58.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:20:58.203 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:58.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:20:58.203 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-09T22:21:03.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:03.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:03.205 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:03.265 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:03.266 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:03.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:03.309+0000 7fb773522780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:03.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:03.310+0000 7fb773522780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:03.326 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:03.327 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:03.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:03.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.756367_365.1", 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:03.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:03.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:03Z" 2024-04-09T22:21:03.332 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:03.333 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.756367_365.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.756367Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:03Z" } 2024-04-09T22:21:03.333 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:03.334 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:03.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:03.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.756367_365.1 2024-04-09T22:21:08.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:08.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:08.335 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:08.395 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:08.395 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:08.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:08.451+0000 7fcf23af1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:08.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:08.452+0000 7fcf23af1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:08.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:08.477 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:08.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.732918_364.1", 2024-04-09T22:21:08.479 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.732918Z", 2024-04-09T22:21:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:08Z" 2024-04-09T22:21:08.483 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:08.485 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.732918_364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.732918Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:08Z" } 2024-04-09T22:21:08.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.732918_364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.732918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:08.485 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:08.485 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.732918_364.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:08.486 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.732918_364.1 2024-04-09T22:21:13.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:13.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:13.487 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:13.545 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:13.545 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:13.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:13.597+0000 7fa90ad79780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:13.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:13.598+0000 7fa90ad79780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:13.617 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:13.617 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:13.617 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:13.617 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:13.617 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:13.617 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:13.618 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:13.619 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.892298_376.1", 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:13Z" 2024-04-09T22:21:13.623 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:13.624 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.892298_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.892298Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:13Z" } 2024-04-09T22:21:13.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.892298_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.892298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:13.625 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:13.625 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.892298_376.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:13.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.892298_376.1 2024-04-09T22:21:18.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:18.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:18.626 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:18.676 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:18.677 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:18.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:18.710+0000 7ff7bdf51780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:18.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:18.711+0000 7ff7bdf51780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:18.734 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:18.734 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:18.734 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:18.734 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:18.734 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:18.734 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:18.735 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:18.736 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.756367_365.1", 2024-04-09T22:21:18.737 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:18.738 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:18.739 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:18.740 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:18.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:18.740 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:18Z" 2024-04-09T22:21:18.740 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:18.741 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.756367_365.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.756367Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:18Z" } 2024-04-09T22:21:18.741 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:18.741 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:18.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:18.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.756367_365.1 2024-04-09T22:21:23.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:23.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:23.743 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:23.799 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:23.799 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:23.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:23.846+0000 7f9c57e10780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:23.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:23.847+0000 7f9c57e10780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:23.865 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:21:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:23Z" 2024-04-09T22:21:23.871 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:23.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:23Z" } 2024-04-09T22:21:23.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:23.873 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:23.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:23.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:21:28.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:28.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:28.874 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:28.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:28.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:28.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:28.972+0000 7fa89bfa9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:28.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:28.974+0000 7fa89bfa9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:28.997 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:28.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:28.999 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:29.000 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:29.001 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:29.002 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:28Z" 2024-04-09T22:21:29.004 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:29.005 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:28Z" } 2024-04-09T22:21:29.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:29.005 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:29.005 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:29.005 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:21:34.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:34.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:34.007 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:34.060 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:34.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:34.105+0000 7fec8e62c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:34.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:34.105+0000 7fec8e62c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:34.129 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:34.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:34.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:34.132 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:34.133 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.780756_368.1", 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:34.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:34.135 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:34Z" 2024-04-09T22:21:34.136 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:34.138 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.780756_368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.780756Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:34Z" } 2024-04-09T22:21:34.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:34.138 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:34.138 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:34.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-09T22:21:39.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:39.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:39.140 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:39.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:39.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:39.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:39.239+0000 7f5fc9b50780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:39.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:39.240+0000 7f5fc9b50780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:39.257 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:39.257 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:39.257 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:39.257 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:39.257 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:39.257 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:39.258 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:39.259 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.780756_368.1", 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:39.260 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:39.261 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:39.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:39.263 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:39Z" 2024-04-09T22:21:39.263 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:39.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.780756_368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.780756Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:39Z" } 2024-04-09T22:21:39.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:39.264 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:39.264 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:39.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-09T22:21:44.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:44.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:44.266 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:44.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:44.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:44.370+0000 7f78774f1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:44.371+0000 7f78774f1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:44.387 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:44.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:44.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:44.390 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.876862_374.1", 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:44.391 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:44.392 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:44Z" 2024-04-09T22:21:44.393 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:44.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.876862_374.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.876862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:44Z" } 2024-04-09T22:21:44.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:44.394 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:44.395 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:44.395 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-09T22:21:49.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:49.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:49.396 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:49.473 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:49.474 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:49.591+0000 7fa3f714e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:49.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:49.592+0000 7fa3f714e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:49.607 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:49.607 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:49.607 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:49.607 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:49.608 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:49.609 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:49.610 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:49.611 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:49.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:49.613 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:49.613 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:49.613 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:49.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:49.613 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:49Z" 2024-04-09T22:21:49.613 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:49.614 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:49Z" } 2024-04-09T22:21:49.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:49.614 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:49.615 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:49.615 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:21:54.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:54.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:54.616 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:54.670 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:54.670 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:54.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:54.710+0000 7f533518c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:54.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:54.711+0000 7f533518c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:54.727 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:54.727 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:54.728 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:54.729 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:54.730 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.876862_374.1", 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:54.731 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:54.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:54.733 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:54.733 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:54.733 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:54.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:54.733 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:54Z" 2024-04-09T22:21:54.733 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:54.734 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.876862_374.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.876862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:54Z" } 2024-04-09T22:21:54.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:54.734 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:54.735 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:54.735 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-09T22:21:59.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:59.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:21:59.736 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:21:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:21:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:21:59.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:59.843+0000 7f2cf5b4e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:59.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:21:59.844+0000 7f2cf5b4e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:21:59.865 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:21:59.865 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:21:59.865 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:59.866 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:59.867 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:21:59.868 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:21:59.869 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:21:59.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:59.871 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:21:59.871 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:21:59.871 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:21:59.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:21:59.871 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:21:59Z" 2024-04-09T22:21:59.871 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:21:59.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:21:59Z" } 2024-04-09T22:21:59.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:21:59.872 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:59.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:21:59.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:22:04.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:04.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:04.875 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:04.932 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:04.932 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:04.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:04.969+0000 7f2668003780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:04.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:04.970+0000 7f2668003780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:04.986 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:04.986 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:04.987 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:04.988 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:04.989 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:04.990 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:04.991 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:04.992 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:04.993 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:04.993 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:04.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:04.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:04.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:04.994 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:04.994 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:04.994 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.889732_375.1", 2024-04-09T22:22:04.994 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:04.994 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:04.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:04.996 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:04.997 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:04.998 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:04.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:04.999 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:04.999 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:04.999 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:04.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:05.000 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:04Z" 2024-04-09T22:22:05.000 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:05.001 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.889732_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.889732Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:04Z" } 2024-04-09T22:22:05.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:05.002 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:05.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:05.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-09T22:22:10.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:10.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:10.004 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:10.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:10.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:10.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:10.100+0000 7fbb57ade780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:10.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:10.101+0000 7fbb57ade780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:10.120 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:10.121 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:10.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:10.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.982479_383.1", 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.982479Z", 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:10.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:10.125 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:10Z" 2024-04-09T22:22:10.126 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:10.127 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.982479_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.982479Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:10Z" } 2024-04-09T22:22:10.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.982479_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.982479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:10.128 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:10.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.982479_383.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:10.128 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.982479_383.1 2024-04-09T22:22:15.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:15.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:15.130 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:15.183 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:15.183 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:15.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:15.216+0000 7f7019403780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:15.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:15.217+0000 7f7019403780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:15.232 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:15.232 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:15.232 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:15.232 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:15.232 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.815777_372.1", 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.815777Z", 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:15.236 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:15.237 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:15.238 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:15.238 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:15.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:15.238 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:15Z" 2024-04-09T22:22:15.238 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:15.239 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.815777_372.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.815777Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:15Z" } 2024-04-09T22:22:15.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.815777_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.815777Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:15.239 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:15.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.815777_372.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:15.239 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.815777_372.1 2024-04-09T22:22:20.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:20.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:20.241 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:20.297 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:20.297 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:20.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:20.350+0000 7fcf27c5a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:20.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:20.351+0000 7fcf27c5a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:20.434 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:20.434 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:20.434 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:20.434 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:20.434 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:20.434 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:20.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:20.436 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:22:20.437 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:20.438 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:20.439 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:20.440 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:20.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:20.440 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:20Z" 2024-04-09T22:22:20.440 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:20.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:20Z" } 2024-04-09T22:22:20.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:20.441 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:20.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:20.441 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:22:25.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:25.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:25.443 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:25.505 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:25.505 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:25.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:25.549+0000 7f02bea54780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:25.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:25.550+0000 7f02bea54780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:25.564 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:25.565 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:25.566 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:25.567 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:25.569 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:25.570 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:25Z" 2024-04-09T22:22:25.571 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:25.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:25Z" } 2024-04-09T22:22:25.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:25.572 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:25.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:25.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:22:30.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:30.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:30.574 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:30.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:30.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:30.671+0000 7f64858ee780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:30.672+0000 7f64858ee780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.876862_374.1", 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-09T22:22:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:30Z" 2024-04-09T22:22:30.693 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:30.694 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.876862_374.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.876862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:30Z" } 2024-04-09T22:22:30.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:30.695 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:30.695 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:30.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-09T22:22:35.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:35.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:35.696 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:35.755 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:35.755 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:35.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:35.869+0000 7fd204f71780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:35.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:35.870+0000 7fd204f71780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:35.912 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:35.913 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:35.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:35.915 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:35.916 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:35.917 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:35.918 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:35Z" 2024-04-09T22:22:35.919 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:35.919 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:35Z" } 2024-04-09T22:22:35.919 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:35.919 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:35.919 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:35.919 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:22:40.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:40.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:40.922 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:41.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:41.024+0000 7f56e5815780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:41.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:41.025+0000 7f56e5815780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:41.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:41.044 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:41.045 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.780756_368.1", 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:41.046 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:41.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:41Z" 2024-04-09T22:22:41.048 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:41.049 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.780756_368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.780756Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:41Z" } 2024-04-09T22:22:41.050 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:41.050 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:41.050 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:41.050 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-09T22:22:46.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:46.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:46.052 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:46.103 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:46.103 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:46.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:46.138+0000 7fcaac735780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:46.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:46.139+0000 7fcaac735780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:46.155 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:46.156 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:46.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:46.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.766836_366.1", 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.766836Z", 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:46.159 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:46.160 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:46Z" 2024-04-09T22:22:46.161 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:46.162 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.766836_366.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.766836Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:46Z" } 2024-04-09T22:22:46.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.766836_366.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.766836Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:46.163 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:46.163 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.766836_366.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:46.163 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.766836_366.1 2024-04-09T22:22:51.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:51.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:51.164 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:51.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:51.278+0000 7efe7ceb3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:51.279+0000 7efe7ceb3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:51.366 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:51.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:51.368 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:51.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:51.368 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:51.368 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:51.368 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:51.369 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:51.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:22:51.371 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:51.372 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:51.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:51.374 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:51.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:51.374 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:51Z" 2024-04-09T22:22:51.374 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:51.375 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:51Z" } 2024-04-09T22:22:51.375 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:51.376 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:51.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:51.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:22:56.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:56.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:22:56.378 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:22:56.431 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:22:56.431 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:22:56.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:56.467+0000 7f962d8c9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:56.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:22:56.468+0000 7f962d8c9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:56.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:56.486 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:56.488 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:22:56.489 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:22:56.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:22:56.491 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:22:56Z" 2024-04-09T22:22:56.491 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:22:56.492 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:22:56Z" } 2024-04-09T22:22:56.492 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:22:56.492 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:56.492 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:22:56.492 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:23:01.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:01.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:01.494 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:01.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:01.595+0000 7f05fc995780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:01.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:01.596+0000 7f05fc995780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:01.612 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:01.612 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:01.613 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:01.614 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.889732_375.1", 2024-04-09T22:23:01.615 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:01.616 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:01.617 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:01.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:01.618 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:01Z" 2024-04-09T22:23:01.618 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:01.619 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.889732_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.889732Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:01Z" } 2024-04-09T22:23:01.619 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:01.619 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:01.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:01.619 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-09T22:23:06.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:06.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:06.621 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:06.679 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:06.679 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:06.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:06.727+0000 7f845fcc9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:06.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:06.728+0000 7f845fcc9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:06.746 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:06.746 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:06.747 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:06.748 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:06.749 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.952757_380.1", 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.952757Z", 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:06.750 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:06.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:06.752 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:06Z" 2024-04-09T22:23:06.752 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:06.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.952757_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.952757Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:06Z" } 2024-04-09T22:23:06.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.952757_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.952757Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:06.753 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:06.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.952757_380.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:06.753 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.952757_380.1 2024-04-09T22:23:11.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:11.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:11.755 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:11.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:11.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:11.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:11.844+0000 7ff9bd753780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:11.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:11.845+0000 7ff9bd753780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:11.860 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:11.860 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:11.861 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:11.862 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:11.863 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:11.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:11.865 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:11Z" 2024-04-09T22:23:11.866 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:11.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:11Z" } 2024-04-09T22:23:11.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:11.867 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:11.868 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:11.868 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:23:16.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:16.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:16.869 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:16.923 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:16.923 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:16.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:16.971+0000 7fc10ca47780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:16.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:16.972+0000 7fc10ca47780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.780756_368.1", 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:16Z" 2024-04-09T22:23:16.996 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:16.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.780756_368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.780756Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:16Z" } 2024-04-09T22:23:16.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:16.997 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:16.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:16.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-09T22:23:21.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:21.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:21.999 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:22.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:22.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:22.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:22.100+0000 7f12ec18c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:22.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:22.101+0000 7f12ec18c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:22.119 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:22.119 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:22.120 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:22.121 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:22.122 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:22.123 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:22.124 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:22.125 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:22.125 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:22.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:22.125 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:22Z" 2024-04-09T22:23:22.125 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:22.126 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:22Z" } 2024-04-09T22:23:22.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:22.127 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:22.127 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:22.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:23:27.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:27.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:27.129 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:27.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:27.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:27.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:27.231+0000 7efedef47780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:27.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:27.232+0000 7efedef47780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:27.249 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:27.250 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:27.250 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:27.250 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:27.253 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:27.254 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:27.254 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:27.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:27.255 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:27.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:27.255 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:27.255 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:27.255 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:27.256 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:27.256 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:27.256 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:27.256 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:27.257 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:27.257 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:27.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:27.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:27.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:27.257 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:27.258 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:27.258 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:27.258 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:27.258 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:27.259 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:27.259 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:27.259 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:27.259 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:27.259 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:27.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:27.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:27.260 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:27.260 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.798600_369.1", 2024-04-09T22:23:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-09T22:23:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:27.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:27.267 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:27Z" 2024-04-09T22:23:27.267 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:27.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.798600_369.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.798600Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:27Z" } 2024-04-09T22:23:27.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:27.269 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:27.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:27.270 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-09T22:23:32.275 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-04-09T22:23:32.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:32.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:32.277 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:32.329 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:32.329 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:32.384 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:23:32.408 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:23:32.408 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.408 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:32.408 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:23:32.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:18:24.029085Z" 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:32.409 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:23:32.410 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:32.410 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:23:32.410 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:32.410 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:23:32.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:32.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:32.410 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:32.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:32.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:32.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:32.513+0000 7f7ae2cc4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:32.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:32.514+0000 7f7ae2cc4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:32.537 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.946918_377.1", 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:32.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:32Z" 2024-04-09T22:23:32.543 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:32.544 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.946918_377.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.946918Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:32Z" } 2024-04-09T22:23:32.544 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.946918_377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.946918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:32.544 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:32.545 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.946918_377.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:32.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.946918_377.1 2024-04-09T22:23:37.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:37.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:37.547 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:37.605 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:37.605 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:37.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:37.651+0000 7efe00d3e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:37.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:37.652+0000 7efe00d3e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:37.690 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:37.691 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:37.692 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:37.693 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:37.694 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:37.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:37.697 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:37Z" 2024-04-09T22:23:37.697 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:37.698 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:37Z" } 2024-04-09T22:23:37.699 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:37.699 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:37.699 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:37.700 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:23:42.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:42.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:42.702 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:42.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:42.755 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:42.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:42.791+0000 7fec78104780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:42.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:42.792+0000 7fec78104780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:42.811 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:42.811 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:42.812 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:42.813 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:42.814 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700425.003891_385.1", 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:05.003891Z", 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:42.815 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:42.816 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:42Z" 2024-04-09T22:23:42.817 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:42.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700425.003891_385.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:05.003891Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:42Z" } 2024-04-09T22:23:42.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700425.003891_385.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:05.003891Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:42.818 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:42.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700425.003891_385.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:42.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700425.003891_385.1 2024-04-09T22:23:47.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:47.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:47.820 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:47.878 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:47.878 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:47.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:47.921+0000 7f306b6f0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:47.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:47.922+0000 7f306b6f0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:47.938 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:47.939 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:47.940 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:47.941 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.766836_366.1", 2024-04-09T22:23:47.942 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.766836Z", 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:47.943 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:47.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:47.945 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:47Z" 2024-04-09T22:23:47.945 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:47.946 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.766836_366.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.766836Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:47Z" } 2024-04-09T22:23:47.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.766836_366.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.766836Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:47.946 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:47.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.766836_366.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:47.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.766836_366.1 2024-04-09T22:23:52.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:52.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:52.947 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:53.004 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:53.005 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:53.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:53.048+0000 7fc135aa4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:53.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:53.049+0000 7fc135aa4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:53.069 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:53Z" 2024-04-09T22:23:53.075 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:53.076 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:53Z" } 2024-04-09T22:23:53.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:53.077 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:53.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:53.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:23:58.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:58.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:23:58.078 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:23:58.130 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:23:58.130 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:23:58.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:58.192+0000 7f3db86e0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:58.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:23:58.193+0000 7f3db86e0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:23:58.208 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:23:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:23:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:23:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:23:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:23:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:23:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:23:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:23:58Z" 2024-04-09T22:23:58.214 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:23:58.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:23:58Z" } 2024-04-09T22:23:58.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:23:58.215 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:58.215 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:23:58.215 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:24:03.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:03.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:03.217 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:03.269 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:03.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:03.321+0000 7f7a5e6e3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:03.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:03.322+0000 7f7a5e6e3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:03.344 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:03.344 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:03.344 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:03.344 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:03.345 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:03.346 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:03.347 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:03.348 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:03Z" 2024-04-09T22:24:03.350 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:03.351 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:03Z" } 2024-04-09T22:24:03.351 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:03.352 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:03.352 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:03.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:24:08.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:08.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:08.353 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:08.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:08.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:08.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:08.464+0000 7f4f2dc3e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:08.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:08.465+0000 7f4f2dc3e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:08Z" 2024-04-09T22:24:08.487 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:08.488 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:08Z" } 2024-04-09T22:24:08.489 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:08.489 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:08.489 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:08.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:24:13.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:13.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:13.491 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:13.553 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:13.553 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:13.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:13.598+0000 7fbc001aa780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:13.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:13.599+0000 7fbc001aa780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:13.619 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:13.620 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:13.621 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:13.622 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:13.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:13.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:13Z" 2024-04-09T22:24:13.625 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:13.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:13Z" } 2024-04-09T22:24:13.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:13.626 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:13.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:13.627 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:24:18.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:18.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:18.628 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:18.678 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:18.678 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:18.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:18.711+0000 7f2c4e6c1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:18.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:18.712+0000 7f2c4e6c1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:18.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:18.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:18.729 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.876862_374.1", 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-09T22:24:18.730 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:18.731 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:18Z" 2024-04-09T22:24:18.732 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:18.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.876862_374.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.876862Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:18Z" } 2024-04-09T22:24:18.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:18.734 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:18.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:18.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-09T22:24:23.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:23.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:23.735 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:23.789 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:23.790 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:23.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:23.843+0000 7f8a1a074780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:23.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:23.844+0000 7f8a1a074780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:23.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:23.866 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:23.867 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:24:23.868 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:23Z" 2024-04-09T22:24:23.871 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:23.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:23Z" } 2024-04-09T22:24:23.872 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:23.872 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:23.872 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:23.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:24:28.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:28.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:28.873 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:28.966 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:29.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:29.042+0000 7f71f49b9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:29.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:29.043+0000 7f71f49b9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:29.058 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:29.058 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:29.059 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:29.060 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:29.061 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.952757_380.1", 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.952757Z", 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:29.062 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:29.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:29.064 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:29.064 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:29.064 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:29.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:29.064 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:29Z" 2024-04-09T22:24:29.064 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:29.065 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.952757_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.952757Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:29Z" } 2024-04-09T22:24:29.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.952757_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.952757Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:29.065 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:29.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.952757_380.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:29.066 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.952757_380.1 2024-04-09T22:24:34.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:34.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:34.067 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:34.120 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:34.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:34.171+0000 7f1b819bc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:34.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:34.172+0000 7f1b819bc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:34.194 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:34.194 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:34.194 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:34.194 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:34.195 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:34.195 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:34.195 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:34.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:34.195 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:34.196 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:34.197 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:34.198 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:34.199 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.952757_380.1", 2024-04-09T22:24:34.199 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:34.199 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:34.199 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:34.199 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.952757Z", 2024-04-09T22:24:34.200 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:34.200 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:34.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:34.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:34.200 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:34.201 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:34.201 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:34.201 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:34.201 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:34.201 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:34.202 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:34.202 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:34.202 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:34.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:34.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:34.203 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:34.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:34.203 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:34.203 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:34.203 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:34.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:34.204 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:34Z" 2024-04-09T22:24:34.204 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:34.206 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.952757_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.952757Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:34Z" } 2024-04-09T22:24:34.206 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.952757_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.952757Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:34.206 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:34.206 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.952757_380.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:34.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.952757_380.1 2024-04-09T22:24:39.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:39.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:39.208 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:39.266 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:39.266 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:39.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:39.317+0000 7f1c6ae82780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:39.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:39.318+0000 7f1c6ae82780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.773250_367.1", 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:39.340 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:39Z" 2024-04-09T22:24:39.340 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:39.341 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.773250_367.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.773250Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:39Z" } 2024-04-09T22:24:39.341 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.773250_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.773250Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:39.341 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:39.341 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.773250_367.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:39.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.773250_367.1 2024-04-09T22:24:44.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:44.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:44.343 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:44.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:44.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:44.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:44.457+0000 7f3d9d07d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:44.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:44.458+0000 7f3d9d07d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:44.533 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.985057_384.1", 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.985057Z", 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:44Z" 2024-04-09T22:24:44.539 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:44.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.985057_384.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.985057Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:44Z" } 2024-04-09T22:24:44.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.985057_384.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.985057Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:44.540 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:44.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.985057_384.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:44.541 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.985057_384.1 2024-04-09T22:24:49.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:49.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:49.542 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:49.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:49.643+0000 7f3e86941780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:49.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:49.644+0000 7f3e86941780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:49.658 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:49.659 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.889732_375.1", 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:49Z" 2024-04-09T22:24:49.664 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:49.665 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.889732_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.889732Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:49Z" } 2024-04-09T22:24:49.665 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:49.666 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:49.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:49.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-09T22:24:54.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:54.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:54.670 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:54.726 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:54.726 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:54.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:54.763+0000 7fd849954780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:54.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:54.764+0000 7fd849954780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:54.780 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:54.781 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.892298_376.1", 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:54Z" 2024-04-09T22:24:54.786 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:54.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.892298_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.892298Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:54Z" } 2024-04-09T22:24:54.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.892298_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.892298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:54.788 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:54.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.892298_376.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:54.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.892298_376.1 2024-04-09T22:24:59.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:59.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:24:59.789 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:24:59.846 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:24:59.846 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:24:59.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:59.896+0000 7f65b1b20780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:59.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:24:59.897+0000 7f65b1b20780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:24:59.916 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:24:59.916 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:24:59.917 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:59.918 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:59.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:59.920 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:24:59.921 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:24:59Z" 2024-04-09T22:24:59.922 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:24:59.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:24:59Z" } 2024-04-09T22:24:59.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:24:59.924 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:59.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:24:59.924 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:25:04.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:04.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:04.925 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:04.977 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:04.977 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:05.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:05.011+0000 7fddd0eab780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:05.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:05.012+0000 7fddd0eab780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:25:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:05Z" 2024-04-09T22:25:05.032 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:05.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:05Z" } 2024-04-09T22:25:05.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:05.034 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:05.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:05.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:25:10.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:10.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:10.035 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:10.091 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:10.091 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:10.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:10.137+0000 7f342ff1b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:10.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:10.138+0000 7f342ff1b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:10.158 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:10.158 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:10.158 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:10.158 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:10.158 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:10.159 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:10.160 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:10.161 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.803140_370.1", 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.803140Z", 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:10.162 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:10.163 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:10.164 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:10.164 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:10.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:10.164 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:10Z" 2024-04-09T22:25:10.164 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:10.165 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.803140_370.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.803140Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:10Z" } 2024-04-09T22:25:10.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.803140_370.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.803140Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:10.165 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:10.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.803140_370.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:10.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.803140_370.1 2024-04-09T22:25:15.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:15.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:15.167 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:15.224 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:15.224 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:15.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:15.278+0000 7f22aba98780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:15.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:15.279+0000 7f22aba98780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:15.299 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:15.300 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:15.301 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:15.302 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.874323_373.1", 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:15.303 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:15.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:15.305 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:15.305 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:15.305 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:15.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:15.305 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:15Z" 2024-04-09T22:25:15.305 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:15.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.874323_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.874323Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:15Z" } 2024-04-09T22:25:15.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:15.306 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:15.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:15.306 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-09T22:25:20.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:20.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:20.308 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:20.360 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:20.360 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:20.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:20.393+0000 7f757497f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:20.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:20.394+0000 7f757497f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:20.409 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:20.409 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:20.410 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:20.411 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.889732_375.1", 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:20.413 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:20.414 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:20.415 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:20.415 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:20.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:20.415 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:20Z" 2024-04-09T22:25:20.415 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:20.416 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.889732_375.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.889732Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:20Z" } 2024-04-09T22:25:20.416 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:20.416 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:20.416 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:20.416 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-09T22:25:25.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:25.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:25.417 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:25.469 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:25.469 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:25.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:25.510+0000 7f3d555f4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:25.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:25.511+0000 7f3d555f4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:25.527 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:25.528 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:25.529 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:25.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:25.530 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:25.530 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:25.530 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:25.530 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:25.531 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:25.531 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:25.531 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:25.531 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:25.531 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:25.532 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:25.532 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:25.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:25.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:25.532 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:25.532 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:25.533 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:25.533 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712700424.813227_371.1", 2024-04-09T22:25:25.533 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:25.533 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:25.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:25.534 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-09T22:25:25.534 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:25.534 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:25.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:25.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:25.535 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:25.535 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:25.535 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:25.535 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:25.535 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:25.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:25.536 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:25.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:25.537 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:25.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:25.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:25.537 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:25.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:25.538 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:25.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:25.538 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:25.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:25.538 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:25Z" 2024-04-09T22:25:25.538 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:25.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712700424.813227_371.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:07:04.813227Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:25Z" } 2024-04-09T22:25:25.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:25.540 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:25.540 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:25.540 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-09T22:25:30.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:30.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:30.541 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:30.597 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:30.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:30.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:30.647+0000 7faf88078780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:30.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:30.648+0000 7faf88078780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:30.669 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:30.669 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:30.669 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:30.669 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:30.669 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:30.670 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:30.671 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:30.672 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:30.673 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:30.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.675 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:30.675 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:30.675 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:30.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.675 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:30Z" 2024-04-09T22:25:30.675 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:30.676 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:30Z" } 2024-04-09T22:25:30.676 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:30.676 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:30.676 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:30.676 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:25:30.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:30.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'BWDQKCT1041WUTW1', '--secret', '0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:30.677 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key BWDQKCT1041WUTW1 --secret 0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:30.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:30.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:30.747 INFO:teuthology.orchestra.run.smithi063.stderr:NOTICE: set zone's realm_id=14ac2a00-44c3-4e52-ac2f-2b97df0062c3 2024-04-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "56298b91-ace3-4c6e-92f5-3baa6b84f7fd", 2024-04-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "remove", 2024-04-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-09T22:25:30.768 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "remove.rgw.control", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "remove.rgw.log", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-09T22:25:30.769 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "remove.rgw.otp", 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "BWDQKCT1041WUTW1", 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T22:25:30.770 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T22:25:30.771 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-09T22:25:30.772 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:30.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:30.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:30.774 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:30.821 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:30.821 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:30.835+0000 7f0bc23fe780 0 period (feef76df-6795-4193-8fee-aa02d98d3595 does not have zone 56298b91-ace3-4c6e-92f5-3baa6b84f7fd configured 2024-04-09T22:25:37.752 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone a5723776-f142-428c-a6df-d7bf949257da 2024-04-09T22:25:37.785 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 6, 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T22:25:37.786 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "56298b91-ace3-4c6e-92f5-3baa6b84f7fd", 2024-04-09T22:25:37.787 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "remove", 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [], 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.788 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.789 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.791 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.792 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:37.793 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T22:25:37.794 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T22:25:37.795 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.796 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.797 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.798 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.799 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.800 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "56298b91-ace3-4c6e-92f5-3baa6b84f7fd", 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 648097763 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.801 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T22:25:37.802 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:37.803 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:37.804 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:37.805 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.806 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T22:25:37.807 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:37.807 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:37.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:37.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:37.808 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:37.893 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:37.894 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:37.895 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:37.896 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:37.896 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:37.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:37.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:37.897 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:37.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:37.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:37.973 INFO:teuthology.orchestra.run.smithi063.stderr:zone name remove not found in zonegroup a 2024-04-09T22:25:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-09T22:25:37.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:37.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:37.979 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:38.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.109 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.176 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.176 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:38.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:38.207+0000 7fc39dc10780 0 WARNING: can't generate connection for zone 56298b91-ace3-4c6e-92f5-3baa6b84f7fd id remove: no endpoints defined 2024-04-09T22:25:38.341 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:38.341 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:38.341 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 7, 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:38.342 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T22:25:38.343 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.344 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.345 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.346 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.346 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.346 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.346 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.346 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T22:25:38.346 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:38.347 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.348 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:38.349 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.350 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.351 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.351 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.351 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.351 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.385 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.386 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.387 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:38.388 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T22:25:38.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.391 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:38.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T22:25:38.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:38.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:38.395 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:38.396 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:38.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-04-09T22:25:38.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.399 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.451 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.451 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:38.487 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e580f898-6ab7-4f4a-bc2f-c166c5759b89", 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "remove_zg", 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "remove_zg", 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [], 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "", 2024-04-09T22:25:38.488 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [], 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [] 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:38.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.490 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:38.490 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.490 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.490 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:38.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.491 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.543 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.543 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:38.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:38.591+0000 7fa6988e6780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "d7b2744b-3c40-42d8-af67-d02b9ee1f8e6", 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "remove", 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout: "control_pool": "remove.rgw.control", 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-09T22:25:38.603 INFO:teuthology.orchestra.run.smithi063.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "log_pool": "remove.rgw.log", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-09T22:25:38.604 INFO:teuthology.orchestra.run.smithi063.stdout: "otp_pool": "remove.rgw.otp", 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "system_key": { 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "access_key": "", 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "secret_key": "" 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_pools": [ 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "default-placement", 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": { 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD": { 2024-04-09T22:25:38.605 INFO:teuthology.orchestra.run.smithi063.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: "index_type": 0, 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: "inline_data": true 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.606 INFO:teuthology.orchestra.run.smithi063.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-09T22:25:38.607 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:38.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.608 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.664 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.664 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:38.764 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:38.764 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:38.764 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 8, 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.765 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.766 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.767 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.768 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.769 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:38.770 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T22:25:38.771 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.772 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.773 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T22:25:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.775 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e580f898-6ab7-4f4a-bc2f-c166c5759b89", 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "remove_zg", 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "remove_zg", 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [], 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "d7b2744b-3c40-42d8-af67-d02b9ee1f8e6", 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.778 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "d7b2744b-3c40-42d8-af67-d02b9ee1f8e6", 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "remove", 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [], 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": false, 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:38.779 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:38.780 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T22:25:38.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "d7b2744b-3c40-42d8-af67-d02b9ee1f8e6", 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 393275986 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:38.783 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T22:25:38.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.784 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:38.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.784 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T22:25:38.784 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:38.784 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T22:25:38.805 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:38.806 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.807 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:38.808 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:38.809 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:38.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.811 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.865 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:38.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:38.895+0000 7fd0ed5ef780 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-04-09T22:25:38.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:38.899+0000 7fd0ed5ef780 0 Removed zone from zonegroup remove_zg 2024-04-09T22:25:38.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:38.922 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:38.973 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:38.973 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:39.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:39.009+0000 7f68ff72f780 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-04-09T22:25:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-09T22:25:39.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:39.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:39.017 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:39.070 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:39.071 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:39.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:39.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T22:25:39.104 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T22:25:39.163 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:39.163 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:39.329 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:39.329 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 9, 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T22:25:39.330 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.331 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:39.332 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:39.333 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:39.334 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:39.335 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:39.336 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:39.337 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T22:25:39.338 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:39.339 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T22:25:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T22:25:39.386 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T22:25:39.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T22:25:39.388 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T22:25:39.389 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T22:25:39.390 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:39.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T22:25:39.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:39.393 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:39.395 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-04-09T22:25:39.395 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-38 2024-04-09T22:25:39.396 DEBUG:boto:path=/xryqqk-38/ 2024-04-09T22:25:39.396 DEBUG:boto:auth_path=/xryqqk-38/ 2024-04-09T22:25:39.396 DEBUG:boto:Method: PUT 2024-04-09T22:25:39.396 DEBUG:boto:Path: /xryqqk-38/ 2024-04-09T22:25:39.396 DEBUG:boto:Data: 2024-04-09T22:25:39.396 DEBUG:boto:Headers: {} 2024-04-09T22:25:39.396 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:25:39.396 DEBUG:boto:Port: 8000 2024-04-09T22:25:39.397 DEBUG:boto:Params: {} 2024-04-09T22:25:39.397 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:25:39.397 DEBUG:boto:Token: None 2024-04-09T22:25:39.398 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:39 GMT /xryqqk-38/ 2024-04-09T22:25:39.398 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LSIGuYbhvjW/6VUF/iBwYfJJPAE= 2024-04-09T22:25:39.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LSIGuYbhvjW/6VUF/iBwYfJJPAE=', 'Content-Length': '0'} 2024-04-09T22:25:40.211 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002919d39918bf22ae-006615c064-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:40.211 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-39 2024-04-09T22:25:40.211 DEBUG:boto:path=/xryqqk-39/ 2024-04-09T22:25:40.212 DEBUG:boto:auth_path=/xryqqk-39/ 2024-04-09T22:25:40.212 DEBUG:boto:Method: PUT 2024-04-09T22:25:40.212 DEBUG:boto:Path: /xryqqk-39/ 2024-04-09T22:25:40.212 DEBUG:boto:Data: 2024-04-09T22:25:40.212 DEBUG:boto:Headers: {} 2024-04-09T22:25:40.212 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:25:40.212 DEBUG:boto:Port: 8001 2024-04-09T22:25:40.212 DEBUG:boto:Params: {} 2024-04-09T22:25:40.212 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:25:40.212 DEBUG:boto:Token: None 2024-04-09T22:25:40.213 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:40 GMT /xryqqk-39/ 2024-04-09T22:25:40.213 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zac2qS0RZ7EW8zR3j1XiN/Vbqtg= 2024-04-09T22:25:40.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zac2qS0RZ7EW8zR3j1XiN/Vbqtg=', 'Content-Length': '0'} 2024-04-09T22:25:40.357 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003117e6196ba84586-006615c064-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:40.358 INFO:rgw_multi.tests:create bucket zone=b1 name=xryqqk-40 2024-04-09T22:25:40.358 DEBUG:boto:path=/xryqqk-40/ 2024-04-09T22:25:40.358 DEBUG:boto:auth_path=/xryqqk-40/ 2024-04-09T22:25:40.358 DEBUG:boto:Method: PUT 2024-04-09T22:25:40.358 DEBUG:boto:Path: /xryqqk-40/ 2024-04-09T22:25:40.358 DEBUG:boto:Data: 2024-04-09T22:25:40.358 DEBUG:boto:Headers: {} 2024-04-09T22:25:40.358 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:25:40.358 DEBUG:boto:Port: 8000 2024-04-09T22:25:40.359 DEBUG:boto:Params: {} 2024-04-09T22:25:40.359 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:25:40.359 DEBUG:boto:Token: None 2024-04-09T22:25:40.359 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:40 GMT /xryqqk-40/ 2024-04-09T22:25:40.359 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SLBQrhvONxZdL5D2OUAvc6Mn2ec= 2024-04-09T22:25:40.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLBQrhvONxZdL5D2OUAvc6Mn2ec=', 'Content-Length': '0'} 2024-04-09T22:25:40.593 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001579421441ddef1a-006615c064-1141-b1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:40.593 INFO:rgw_multi.tests:create bucket zone=b2 name=xryqqk-41 2024-04-09T22:25:40.594 DEBUG:boto:path=/xryqqk-41/ 2024-04-09T22:25:40.594 DEBUG:boto:auth_path=/xryqqk-41/ 2024-04-09T22:25:40.594 DEBUG:boto:Method: PUT 2024-04-09T22:25:40.594 DEBUG:boto:Path: /xryqqk-41/ 2024-04-09T22:25:40.594 DEBUG:boto:Data: 2024-04-09T22:25:40.594 DEBUG:boto:Headers: {} 2024-04-09T22:25:40.594 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:25:40.594 DEBUG:boto:Port: 8001 2024-04-09T22:25:40.594 DEBUG:boto:Params: {} 2024-04-09T22:25:40.595 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:25:40.595 DEBUG:boto:Token: None 2024-04-09T22:25:40.595 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:40 GMT /xryqqk-41/ 2024-04-09T22:25:40.595 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:j65qT/pXQLBscbUojJPhouYy3j4= 2024-04-09T22:25:40.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j65qT/pXQLBscbUojJPhouYy3j4=', 'Content-Length': '0'} 2024-04-09T22:25:40.995 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007e7e2d20a192b888-006615c064-1145-b2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:40.995 DEBUG:boto:path=/xryqqk-38/ 2024-04-09T22:25:40.996 DEBUG:boto:auth_path=/xryqqk-38/ 2024-04-09T22:25:40.996 DEBUG:boto:path=/xryqqk-38/?website 2024-04-09T22:25:40.996 DEBUG:boto:auth_path=/xryqqk-38/?website 2024-04-09T22:25:40.996 DEBUG:boto:Method: PUT 2024-04-09T22:25:40.996 DEBUG:boto:Path: /xryqqk-38/?website 2024-04-09T22:25:40.996 DEBUG:boto:Data: index.htmlerror.html 2024-04-09T22:25:40.996 DEBUG:boto:Headers: {} 2024-04-09T22:25:40.996 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:25:40.996 DEBUG:boto:Port: 8000 2024-04-09T22:25:40.996 DEBUG:boto:Params: {} 2024-04-09T22:25:40.997 DEBUG:boto:Token: None 2024-04-09T22:25:40.997 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:40 GMT /xryqqk-38/?website 2024-04-09T22:25:40.997 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4uBebm/Ubj3piyklNVdPqF2pGdQ= 2024-04-09T22:25:40.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4uBebm/Ubj3piyklNVdPqF2pGdQ=', 'Content-Length': '241'} 2024-04-09T22:25:40.998 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000fa6f6c6868b2a2b9-006615c064-1b6d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:25:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:40.999 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-04-09T22:25:41.000 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-42 2024-04-09T22:25:41.000 DEBUG:boto:path=/xryqqk-42/ 2024-04-09T22:25:41.000 DEBUG:boto:auth_path=/xryqqk-42/ 2024-04-09T22:25:41.000 DEBUG:boto:Method: PUT 2024-04-09T22:25:41.000 DEBUG:boto:Path: /xryqqk-42/ 2024-04-09T22:25:41.000 DEBUG:boto:Data: 2024-04-09T22:25:41.000 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.000 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:25:41.000 DEBUG:boto:Port: 8000 2024-04-09T22:25:41.001 DEBUG:boto:Params: {} 2024-04-09T22:25:41.001 DEBUG:boto:Token: None 2024-04-09T22:25:41.001 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-42/ 2024-04-09T22:25:41.001 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rbQEm4jzcGqbCC9XOpWiylQMZiM= 2024-04-09T22:25:41.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rbQEm4jzcGqbCC9XOpWiylQMZiM=', 'Content-Length': '0'} 2024-04-09T22:25:41.015 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c971cb46f5ed76a4-006615c065-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:41.016 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-43 2024-04-09T22:25:41.016 DEBUG:boto:path=/xryqqk-43/ 2024-04-09T22:25:41.016 DEBUG:boto:auth_path=/xryqqk-43/ 2024-04-09T22:25:41.016 DEBUG:boto:Method: PUT 2024-04-09T22:25:41.016 DEBUG:boto:Path: /xryqqk-43/ 2024-04-09T22:25:41.016 DEBUG:boto:Data: 2024-04-09T22:25:41.016 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.016 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:25:41.016 DEBUG:boto:Port: 8001 2024-04-09T22:25:41.017 DEBUG:boto:Params: {} 2024-04-09T22:25:41.017 DEBUG:boto:Token: None 2024-04-09T22:25:41.017 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-43/ 2024-04-09T22:25:41.017 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZVJUZi9pnx3r5BKbqEnWGMjzr1Q= 2024-04-09T22:25:41.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZVJUZi9pnx3r5BKbqEnWGMjzr1Q=', 'Content-Length': '0'} 2024-04-09T22:25:41.051 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e1962d168e7f4588-006615c065-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:41.051 INFO:rgw_multi.tests:create bucket zone=b1 name=xryqqk-44 2024-04-09T22:25:41.051 DEBUG:boto:path=/xryqqk-44/ 2024-04-09T22:25:41.051 DEBUG:boto:auth_path=/xryqqk-44/ 2024-04-09T22:25:41.051 DEBUG:boto:Method: PUT 2024-04-09T22:25:41.051 DEBUG:boto:Path: /xryqqk-44/ 2024-04-09T22:25:41.051 DEBUG:boto:Data: 2024-04-09T22:25:41.051 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.051 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:25:41.052 DEBUG:boto:Port: 8000 2024-04-09T22:25:41.052 DEBUG:boto:Params: {} 2024-04-09T22:25:41.052 DEBUG:boto:Token: None 2024-04-09T22:25:41.052 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-44/ 2024-04-09T22:25:41.052 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rzbj3n8/qFvZ9NcZeIlFXbLgrS8= 2024-04-09T22:25:41.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rzbj3n8/qFvZ9NcZeIlFXbLgrS8=', 'Content-Length': '0'} 2024-04-09T22:25:41.504 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d63ed0cf1305b515-006615c065-1149-b1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:41.505 INFO:rgw_multi.tests:create bucket zone=b2 name=xryqqk-45 2024-04-09T22:25:41.505 DEBUG:boto:path=/xryqqk-45/ 2024-04-09T22:25:41.505 DEBUG:boto:auth_path=/xryqqk-45/ 2024-04-09T22:25:41.505 DEBUG:boto:Method: PUT 2024-04-09T22:25:41.505 DEBUG:boto:Path: /xryqqk-45/ 2024-04-09T22:25:41.505 DEBUG:boto:Data: 2024-04-09T22:25:41.505 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.505 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:25:41.505 DEBUG:boto:Port: 8001 2024-04-09T22:25:41.505 DEBUG:boto:Params: {} 2024-04-09T22:25:41.506 DEBUG:boto:Token: None 2024-04-09T22:25:41.506 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-45/ 2024-04-09T22:25:41.506 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uiEfljr4GNeU5zZybZ7ltH7+Phk= 2024-04-09T22:25:41.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uiEfljr4GNeU5zZybZ7ltH7+Phk=', 'Content-Length': '0'} 2024-04-09T22:25:41.565 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f74f20767a7fd590-006615c065-1145-b2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:41.566 DEBUG:boto:path=/xryqqk-42/ 2024-04-09T22:25:41.566 DEBUG:boto:auth_path=/xryqqk-42/ 2024-04-09T22:25:41.566 DEBUG:boto:path=/xryqqk-42/?policy 2024-04-09T22:25:41.566 DEBUG:boto:auth_path=/xryqqk-42/?policy 2024-04-09T22:25:41.566 DEBUG:boto:Method: PUT 2024-04-09T22:25:41.567 DEBUG:boto:Path: /xryqqk-42/?policy 2024-04-09T22:25:41.567 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-09T22:25:41.567 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.567 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:25:41.567 DEBUG:boto:Port: 8000 2024-04-09T22:25:41.567 DEBUG:boto:Params: {} 2024-04-09T22:25:41.567 DEBUG:boto:Token: None 2024-04-09T22:25:41.567 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-42/?policy 2024-04-09T22:25:41.568 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mhptCsCNr0gZFUp9uHC6MP0Lahg= 2024-04-09T22:25:41.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mhptCsCNr0gZFUp9uHC6MP0Lahg=', 'Content-Length': '97'} 2024-04-09T22:25:41.575 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002fff2cc3f3abf12f-006615c065-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:41.575 DEBUG:boto:path=/xryqqk-42/ 2024-04-09T22:25:41.575 DEBUG:boto:auth_path=/xryqqk-42/ 2024-04-09T22:25:41.575 DEBUG:boto:path=/xryqqk-42/?policy 2024-04-09T22:25:41.575 DEBUG:boto:auth_path=/xryqqk-42/?policy 2024-04-09T22:25:41.575 DEBUG:boto:Method: GET 2024-04-09T22:25:41.575 DEBUG:boto:Path: /xryqqk-42/?policy 2024-04-09T22:25:41.575 DEBUG:boto:Data: 2024-04-09T22:25:41.576 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.576 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:25:41.576 DEBUG:boto:Port: 8000 2024-04-09T22:25:41.576 DEBUG:boto:Params: {} 2024-04-09T22:25:41.576 DEBUG:boto:Token: None 2024-04-09T22:25:41.576 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-42/?policy 2024-04-09T22:25:41.576 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qMZiMJKnDP0yMSiNs/VFXdEJKoI= 2024-04-09T22:25:41.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qMZiMJKnDP0yMSiNs/VFXdEJKoI=', 'Content-Length': '0'} 2024-04-09T22:25:41.578 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001358b4d9fc302a82-006615c065-1b6d-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 09 Apr 2024 22:25:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:41.578 DEBUG:boto:path=/xryqqk-43/ 2024-04-09T22:25:41.578 DEBUG:boto:auth_path=/xryqqk-43/ 2024-04-09T22:25:41.578 DEBUG:boto:path=/xryqqk-43/?policy 2024-04-09T22:25:41.578 DEBUG:boto:auth_path=/xryqqk-43/?policy 2024-04-09T22:25:41.578 DEBUG:boto:Method: PUT 2024-04-09T22:25:41.579 DEBUG:boto:Path: /xryqqk-43/?policy 2024-04-09T22:25:41.579 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-09T22:25:41.579 DEBUG:boto:Headers: {} 2024-04-09T22:25:41.579 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:25:41.579 DEBUG:boto:Port: 8001 2024-04-09T22:25:41.579 DEBUG:boto:Params: {} 2024-04-09T22:25:41.579 DEBUG:boto:Token: None 2024-04-09T22:25:41.579 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:41 GMT /xryqqk-43/?policy 2024-04-09T22:25:41.580 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1c3ViCV5nJ/4//kgCSatQ/xtt6U= 2024-04-09T22:25:41.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1c3ViCV5nJ/4//kgCSatQ/xtt6U=', 'Content-Length': '97'} 2024-04-09T22:25:42.592 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086afb6ccd957d24e-006615c065-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:42.592 DEBUG:boto:path=/xryqqk-43/ 2024-04-09T22:25:42.592 DEBUG:boto:auth_path=/xryqqk-43/ 2024-04-09T22:25:42.593 DEBUG:boto:path=/xryqqk-43/?policy 2024-04-09T22:25:42.593 DEBUG:boto:auth_path=/xryqqk-43/?policy 2024-04-09T22:25:42.593 DEBUG:boto:Method: GET 2024-04-09T22:25:42.593 DEBUG:boto:Path: /xryqqk-43/?policy 2024-04-09T22:25:42.593 DEBUG:boto:Data: 2024-04-09T22:25:42.593 DEBUG:boto:Headers: {} 2024-04-09T22:25:42.593 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:25:42.593 DEBUG:boto:Port: 8001 2024-04-09T22:25:42.593 DEBUG:boto:Params: {} 2024-04-09T22:25:42.593 DEBUG:boto:Token: None 2024-04-09T22:25:42.594 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:25:42 GMT /xryqqk-43/?policy 2024-04-09T22:25:42.594 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EfLmDlVP/wmIHljjDA6UCboYqho= 2024-04-09T22:25:42.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EfLmDlVP/wmIHljjDA6UCboYqho=', 'Content-Length': '0'} 2024-04-09T22:25:42.595 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a26a3b47cf0d6cf3-006615c066-1b6f-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 09 Apr 2024 22:25:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:42.595 DEBUG:boto:path=/xryqqk-44/ 2024-04-09T22:25:42.595 DEBUG:boto:auth_path=/xryqqk-44/ 2024-04-09T22:25:42.595 DEBUG:boto:path=/xryqqk-44/?policy 2024-04-09T22:25:42.596 DEBUG:boto:auth_path=/xryqqk-44/?policy 2024-04-09T22:25:42.596 DEBUG:boto:Method: PUT 2024-04-09T22:25:42.596 DEBUG:boto:Path: /xryqqk-44/?policy 2024-04-09T22:25:42.596 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-09T22:25:42.596 DEBUG:boto:Headers: {} 2024-04-09T22:25:42.596 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:25:42.596 DEBUG:boto:Port: 8000 2024-04-09T22:25:42.596 DEBUG:boto:Params: {} 2024-04-09T22:25:42.596 DEBUG:boto:Token: None 2024-04-09T22:25:42.596 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:42 GMT /xryqqk-44/?policy 2024-04-09T22:25:42.597 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c1lVEExDSFFKdHkx+pyrhPhvjfo= 2024-04-09T22:25:42.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c1lVEExDSFFKdHkx+pyrhPhvjfo=', 'Content-Length': '97'} 2024-04-09T22:25:43.609 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bb165dea8688f260-006615c066-1149-b1'), ('Date', 'Tue, 09 Apr 2024 22:25:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:43.609 DEBUG:boto:path=/xryqqk-44/ 2024-04-09T22:25:43.609 DEBUG:boto:auth_path=/xryqqk-44/ 2024-04-09T22:25:43.610 DEBUG:boto:path=/xryqqk-44/?policy 2024-04-09T22:25:43.610 DEBUG:boto:auth_path=/xryqqk-44/?policy 2024-04-09T22:25:43.610 DEBUG:boto:Method: GET 2024-04-09T22:25:43.610 DEBUG:boto:Path: /xryqqk-44/?policy 2024-04-09T22:25:43.610 DEBUG:boto:Data: 2024-04-09T22:25:43.610 DEBUG:boto:Headers: {} 2024-04-09T22:25:43.610 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8000 2024-04-09T22:25:43.610 DEBUG:boto:Port: 8000 2024-04-09T22:25:43.610 DEBUG:boto:Params: {} 2024-04-09T22:25:43.610 DEBUG:boto:Token: None 2024-04-09T22:25:43.611 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:25:43 GMT /xryqqk-44/?policy 2024-04-09T22:25:43.611 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:N+vgmwzFaDRL9yoFQOsF/xxhXgQ= 2024-04-09T22:25:43.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:N+vgmwzFaDRL9yoFQOsF/xxhXgQ=', 'Content-Length': '0'} 2024-04-09T22:25:43.612 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008b2a3e8cd83a960c-006615c067-1149-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 09 Apr 2024 22:25:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:43.612 DEBUG:boto:path=/xryqqk-45/ 2024-04-09T22:25:43.612 DEBUG:boto:auth_path=/xryqqk-45/ 2024-04-09T22:25:43.613 DEBUG:boto:path=/xryqqk-45/?policy 2024-04-09T22:25:43.613 DEBUG:boto:auth_path=/xryqqk-45/?policy 2024-04-09T22:25:43.613 DEBUG:boto:Method: PUT 2024-04-09T22:25:43.613 DEBUG:boto:Path: /xryqqk-45/?policy 2024-04-09T22:25:43.613 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-09T22:25:43.613 DEBUG:boto:Headers: {} 2024-04-09T22:25:43.613 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:25:43.613 DEBUG:boto:Port: 8001 2024-04-09T22:25:43.613 DEBUG:boto:Params: {} 2024-04-09T22:25:43.613 DEBUG:boto:Token: None 2024-04-09T22:25:43.614 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:43 GMT /xryqqk-45/?policy 2024-04-09T22:25:43.614 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:O0g/A110TjHs53+ReEXJx0fIAEI= 2024-04-09T22:25:43.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:O0g/A110TjHs53+ReEXJx0fIAEI=', 'Content-Length': '97'} 2024-04-09T22:25:44.627 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c85c832b41e56a6-006615c067-1145-b2'), ('Date', 'Tue, 09 Apr 2024 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:44.627 DEBUG:boto:path=/xryqqk-45/ 2024-04-09T22:25:44.627 DEBUG:boto:auth_path=/xryqqk-45/ 2024-04-09T22:25:44.627 DEBUG:boto:path=/xryqqk-45/?policy 2024-04-09T22:25:44.627 DEBUG:boto:auth_path=/xryqqk-45/?policy 2024-04-09T22:25:44.627 DEBUG:boto:Method: GET 2024-04-09T22:25:44.627 DEBUG:boto:Path: /xryqqk-45/?policy 2024-04-09T22:25:44.628 DEBUG:boto:Data: 2024-04-09T22:25:44.628 DEBUG:boto:Headers: {} 2024-04-09T22:25:44.628 DEBUG:boto:Host: smithi137.front.sepia.ceph.com:8001 2024-04-09T22:25:44.628 DEBUG:boto:Port: 8001 2024-04-09T22:25:44.628 DEBUG:boto:Params: {} 2024-04-09T22:25:44.628 DEBUG:boto:Token: None 2024-04-09T22:25:44.628 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:25:44 GMT /xryqqk-45/?policy 2024-04-09T22:25:44.628 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OLdT9o63+aUtOC8NkE0OwkKlzSE= 2024-04-09T22:25:44.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OLdT9o63+aUtOC8NkE0OwkKlzSE=', 'Content-Length': '0'} 2024-04-09T22:25:44.630 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006cf5db2e4e3bdef3-006615c068-1145-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Tue, 09 Apr 2024 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:44.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-04-09T22:25:44.631 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-46 2024-04-09T22:25:44.631 DEBUG:boto:path=/xryqqk-46/ 2024-04-09T22:25:44.631 DEBUG:boto:auth_path=/xryqqk-46/ 2024-04-09T22:25:44.631 DEBUG:boto:Method: PUT 2024-04-09T22:25:44.631 DEBUG:boto:Path: /xryqqk-46/ 2024-04-09T22:25:44.632 DEBUG:boto:Data: 2024-04-09T22:25:44.632 DEBUG:boto:Headers: {} 2024-04-09T22:25:44.632 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:25:44.632 DEBUG:boto:Port: 8000 2024-04-09T22:25:44.632 DEBUG:boto:Params: {} 2024-04-09T22:25:44.632 DEBUG:boto:Token: None 2024-04-09T22:25:44.632 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:44 GMT /xryqqk-46/ 2024-04-09T22:25:44.632 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MxrmYYaRGkGRdxEx3GjDDNuuprk= 2024-04-09T22:25:44.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MxrmYYaRGkGRdxEx3GjDDNuuprk=', 'Content-Length': '0'} 2024-04-09T22:25:44.656 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000058cdd5c07fe2acd4-006615c068-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:44.656 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-47 2024-04-09T22:25:44.656 DEBUG:boto:path=/xryqqk-47/ 2024-04-09T22:25:44.656 DEBUG:boto:auth_path=/xryqqk-47/ 2024-04-09T22:25:44.656 DEBUG:boto:Method: PUT 2024-04-09T22:25:44.656 DEBUG:boto:Path: /xryqqk-47/ 2024-04-09T22:25:44.656 DEBUG:boto:Data: 2024-04-09T22:25:44.657 DEBUG:boto:Headers: {} 2024-04-09T22:25:44.657 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:25:44.657 DEBUG:boto:Port: 8001 2024-04-09T22:25:44.657 DEBUG:boto:Params: {} 2024-04-09T22:25:44.657 DEBUG:boto:Token: None 2024-04-09T22:25:44.657 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:25:44 GMT /xryqqk-47/ 2024-04-09T22:25:44.657 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:igcA+CdhSC69t/VPxDHOeyGG2Ow= 2024-04-09T22:25:44.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:igcA+CdhSC69t/VPxDHOeyGG2Ow=', 'Content-Length': '0'} 2024-04-09T22:25:44.693 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad9bef747a9e9471-006615c068-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:25:44.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:44.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:44.694 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:44.747 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:44.747 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701544.676887_623.1", 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:25:44.676887Z" 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:25:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:44.813 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:25:44.814 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:44.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:25:44.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:44.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:44.814 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:44.863 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:44.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:44.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:44.917+0000 7f8a4955e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:44.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:44.918+0000 7f8a4955e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.941 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:44.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:44.943 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:44.944 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:44.945 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:44Z" 2024-04-09T22:25:44.946 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:44.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:44Z" } 2024-04-09T22:25:44.947 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:44.948 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:44.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:44.948 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:25:49.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:49.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:49.950 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:50.008 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:50.008 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:50.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:50.046+0000 7fb845c69780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:50.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:50.047+0000 7fb845c69780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:50.063 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:50.064 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:50.065 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:50.066 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:50.067 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:50.068 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:50.069 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:50.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:50.069 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:50Z" 2024-04-09T22:25:50.069 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:50.070 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:50Z" } 2024-04-09T22:25:50.070 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:50.070 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:50.070 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:50.071 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:25:55.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:55.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:25:55.072 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:25:55.132 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:25:55.132 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:25:55.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:55.181+0000 7f98a5315780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:55.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:25:55.182+0000 7f98a5315780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:25:55.206 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:55.207 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:55.208 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:25:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:25:55.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:55.212 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:25:55.212 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:25:55.212 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:25:55.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:25:55.212 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:25:55Z" 2024-04-09T22:25:55.212 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:25:55.214 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:25:55Z" } 2024-04-09T22:25:55.214 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:25:55.214 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:55.214 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:25:55.214 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:00.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:00.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:00.216 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:00.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:00.322+0000 7f71d9819780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:00.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:00.323+0000 7f71d9819780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:00.339 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:00.340 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:00.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:00.342 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:00Z" 2024-04-09T22:26:00.344 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:00.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:00Z" } 2024-04-09T22:26:00.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:00.345 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:00.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:00.346 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:05.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:05.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:05.347 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:05.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:05.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:05.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:05.437+0000 7f5551a2b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:05.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:05.438+0000 7f5551a2b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:05.453 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:05.453 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:05.453 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:05.453 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:05.453 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:05.454 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:05.455 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:05.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:05.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:05.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:05.459 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:05Z" 2024-04-09T22:26:05.459 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:05.459 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:05Z" } 2024-04-09T22:26:05.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:05.461 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:05.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:05.461 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:10.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:10.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:10.463 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:10.523 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:10.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:10.565+0000 7fa92e5bd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:10.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:10.566+0000 7fa92e5bd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:10.586 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:10.586 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:10.586 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:10.586 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:10.587 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:10.588 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:10.589 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:10.590 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:10.591 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:10Z" 2024-04-09T22:26:10.592 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:10.592 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:10Z" } 2024-04-09T22:26:10.593 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:10.593 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:10.593 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:10.593 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:15.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:15.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:15.594 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:15.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:15.695+0000 7fe3d87a5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:15.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:15.696+0000 7fe3d87a5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:15Z" 2024-04-09T22:26:15.720 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:15.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:15Z" } 2024-04-09T22:26:15.721 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:15.722 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:15.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:15.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:20.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:20.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:20.723 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:20.778 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:20.779 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:20.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:20.830+0000 7f3e63af9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:20.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:20.831+0000 7f3e63af9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:20.853 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:20.853 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:20.853 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:20.853 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:20.853 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:20.854 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:20.855 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:20.856 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:20.857 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:20.858 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:20Z" 2024-04-09T22:26:20.859 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:20.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:20Z" } 2024-04-09T22:26:20.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:20.860 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:20.860 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:20.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:25.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:25.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:25.861 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:25.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:25.921 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:25.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:25.960+0000 7f256e6de780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:25.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:25.961+0000 7f256e6de780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:25.976 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:25.976 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:25.976 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:25.976 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:25.976 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:25.976 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:25.977 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:25.978 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:25.979 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:25.980 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:25.981 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:25.981 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:25.981 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:25.981 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:25.981 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:25Z" 2024-04-09T22:26:25.982 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:25.983 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:25Z" } 2024-04-09T22:26:25.984 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:25.984 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:25.984 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:25.984 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:30.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:30.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:30.985 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:31.042 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:31.043 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:31.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:31.086+0000 7f78ec131780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:31.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:31.087+0000 7f78ec131780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:31.101 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:31.102 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:31.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:31.104 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:31.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:31Z" 2024-04-09T22:26:31.107 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:31.108 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:31Z" } 2024-04-09T22:26:31.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:31.108 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:31.108 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:31.109 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:36.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:36.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:36.110 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:36.169 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:36.169 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:36.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:36.211+0000 7fb81e600780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:36.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:36.212+0000 7fb81e600780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:36.235 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:36.235 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:36.235 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:36.235 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:36.235 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:36.236 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:36.237 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:36.238 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:36.239 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:36.240 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:36Z" 2024-04-09T22:26:36.241 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:36.242 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:36Z" } 2024-04-09T22:26:36.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:36.242 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:36.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:36.243 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:41.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:41.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:41.245 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:41.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:41.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:41.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:41.351+0000 7f7ee9f56780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:41.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:41.352+0000 7f7ee9f56780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:41.369 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:41.369 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:41.369 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:41.370 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:41.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:41.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:41.373 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:41.374 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:41.375 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:41.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:41.376 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:41Z" 2024-04-09T22:26:41.376 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:41.377 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:41Z" } 2024-04-09T22:26:41.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:41.377 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:41.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:41.377 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:46.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:46.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:46.379 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:46.436 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:46.436 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:46.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:46.474+0000 7fcc3e854780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:46.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:46.475+0000 7fcc3e854780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:46.491 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:46.491 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:46.492 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:46.493 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:46.494 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:46.495 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:46.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:46.497 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:46Z" 2024-04-09T22:26:46.497 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:46.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:46Z" } 2024-04-09T22:26:46.498 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:46.498 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:46.498 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:46.498 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:51.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:51.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:51.500 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:51.557 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:51.557 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:51.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:51.601+0000 7f0dcfedb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:51.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:51.602+0000 7f0dcfedb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:51.623 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:51.623 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:51.623 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:51.623 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:51.624 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:51.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:51.626 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:51.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:51.628 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:51Z" 2024-04-09T22:26:51.629 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:51.630 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:51Z" } 2024-04-09T22:26:51.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:51.631 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:51.631 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:51.631 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:26:56.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:56.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:26:56.632 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:26:56.689 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:26:56.689 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:26:56.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:56.743+0000 7fd25d769780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:56.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:26:56.744+0000 7fd25d769780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:26:56.763 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:26:56.763 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:26:56.763 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:26:56.763 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:26:56.763 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:26:56.763 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:26:56.764 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:56.765 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:26:56.766 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:26:56.767 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:26:56.768 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:26:56.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:26:56.769 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:26:56Z" 2024-04-09T22:26:56.769 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:26:56.770 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:26:56Z" } 2024-04-09T22:26:56.770 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:26:56.770 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:56.770 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:26:56.770 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:01.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:01.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:01.772 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:01.828 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:01.829 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:01.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:01.865+0000 7fde9cad8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:01.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:01.866+0000 7fde9cad8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:01.882 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:01.882 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:01.882 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:01.882 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:01.882 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:01.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:01.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:01.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:01.886 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:01.887 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:01.888 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:01.888 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:01.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:01.888 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:01Z" 2024-04-09T22:27:01.888 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:01.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:01Z" } 2024-04-09T22:27:01.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:01.889 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:01.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:01.889 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:06.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:06.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:06.890 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:06.949 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:06.950 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:06.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:06.988+0000 7f742cbf7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:06.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:06.989+0000 7f742cbf7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:07.005 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:07.006 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:07.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:07.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:07.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:06Z" 2024-04-09T22:27:07.011 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:07.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:06Z" } 2024-04-09T22:27:07.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:07.013 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:07.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:07.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:11.260 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:27:11.260+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:27:12.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:12.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:12.014 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:12.071 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:12.071 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:12.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:12.108+0000 7fc79ca1e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:12.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:12.109+0000 7fc79ca1e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:12.130 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:12.130 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:12.131 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:12.132 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:12.133 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:12.134 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:12.135 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:12.136 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:12Z" 2024-04-09T22:27:12.136 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:12.137 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:12Z" } 2024-04-09T22:27:12.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:12.137 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:12.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:12.137 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:17.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:17.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:17.139 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:17.197 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:17.197 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:17.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:17.242+0000 7f6e85c96780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:17.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:17.243+0000 7f6e85c96780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:17Z" 2024-04-09T22:27:17.266 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:17.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:17Z" } 2024-04-09T22:27:17.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:17.267 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:17.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:17.268 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:22.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:22.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:22.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:22.325 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:22.326 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:22.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:22.365+0000 7f966938b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:22.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:22.367+0000 7f966938b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:22.382 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:22.382 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:22.382 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:22.382 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:22.382 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:22.382 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:22.383 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:22.384 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:22.385 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:22.386 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:22.387 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:22.388 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:22.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:22.388 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:22Z" 2024-04-09T22:27:22.388 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:22.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:22Z" } 2024-04-09T22:27:22.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:22.389 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:22.389 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:22.389 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:27.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:27.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:27.391 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:27.449 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:27.449 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:27.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:27.487+0000 7fc84b776780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:27.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:27.488+0000 7fc84b776780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:27.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:27.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:27.506 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:27.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:27.508 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:27.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:27.510 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:27Z" 2024-04-09T22:27:27.510 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:27.511 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:27Z" } 2024-04-09T22:27:27.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:27.511 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:27.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:27.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:32.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:32.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:32.513 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:32.571 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:32.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:32.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:32.625+0000 7f304a8c3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:32.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:32.626+0000 7f304a8c3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:32.649 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:32.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:32.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:32.652 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:32.653 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:32Z" 2024-04-09T22:27:32.654 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:32.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:32Z" } 2024-04-09T22:27:32.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:32.656 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:32.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:32.656 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:37.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:37.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:37.658 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:37.715 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:37.716 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:37.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:37.751+0000 7fdba9a67780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:37.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:37.752+0000 7fdba9a67780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:37.773 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:37.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:37.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:37.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701104.029085_507.1", 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:37.777 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:37.778 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:37Z" 2024-04-09T22:27:37.779 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:37.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701104.029085_507.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:18:24.029085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:37Z" } 2024-04-09T22:27:37.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:37.781 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:37.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:37.781 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-09T22:27:42.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:42.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:42.783 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:42.849 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:42.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:42.900+0000 7f302c7a1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:42.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:42.902+0000 7f302c7a1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:42.918 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:42.918 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:42.918 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:42.918 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:42.919 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:42.920 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:42.921 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:42.922 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:42Z" 2024-04-09T22:27:42.923 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:42.924 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:42Z" } 2024-04-09T22:27:42.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:42.925 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:42.925 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:42.925 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-09T22:27:47.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:47.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:47.926 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:47.983 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:47.983 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:48.019+0000 7f7ddfb5b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:48.020+0000 7f7ddfb5b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:48.036 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:48.036 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:48.036 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:48.036 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:48.036 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:48.036 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:48.037 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:48.038 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:48.039 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.265896_573.1", 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:48.040 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:48.041 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:48.042 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:48.042 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:48.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:48.042 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:48Z" 2024-04-09T22:27:48.042 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:48.043 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.265896_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.265896Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:48Z" } 2024-04-09T22:27:48.043 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:48.043 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:48.043 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:48.043 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.265896_573.1 2024-04-09T22:27:53.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:53.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:53.045 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:53.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:53.153+0000 7f4b6c52b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:53.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:53.154+0000 7f4b6c52b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:53.175 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:53.175 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:53.175 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:53.175 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:53.176 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:53.177 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:53.178 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.265896_573.1", 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:53.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:53.180 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:53Z" 2024-04-09T22:27:53.181 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:53.182 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.265896_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.265896Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:53Z" } 2024-04-09T22:27:53.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:53.182 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:53.183 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:53.183 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.265896_573.1 2024-04-09T22:27:58.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:58.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:27:58.184 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:27:58.276 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:27:58.276 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:27:58.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:58.314+0000 7fb021a15780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:58.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:27:58.315+0000 7fb021a15780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:27:58.331 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:58.332 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:58.333 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:27:58.334 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:27:58.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:58.336 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:27:58.336 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:27:58.336 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:27:58.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:27:58.336 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:27:58Z" 2024-04-09T22:27:58.336 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:27:58.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:27:58Z" } 2024-04-09T22:27:58.337 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:27:58.337 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:58.337 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:27:58.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:28:03.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:03.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:03.339 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:03.391 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:03.391 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:03.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:03.429+0000 7fea058c4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:03.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:03.430+0000 7fea058c4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:03.449 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:03.449 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:03.450 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:03.451 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:03.452 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:03.453 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:03.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:03Z" 2024-04-09T22:28:03.455 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:03.456 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:03Z" } 2024-04-09T22:28:03.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:03.456 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:03.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:03.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:28:08.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:08.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:08.459 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:08.509 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:08.509 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:08.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:08.560+0000 7f3e7aed7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:08.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:08.560+0000 7f3e7aed7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:08.583 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:08.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:28:08.585 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:08.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:08Z" 2024-04-09T22:28:08.588 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:08.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:08Z" } 2024-04-09T22:28:08.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:08.589 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:08.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:08.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:28:13.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:13.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:13.590 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:13.641 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:13.642 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:13.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:13.675+0000 7f0067dad780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:13.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:13.676+0000 7f0067dad780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:13.695 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:13.695 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:13.695 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:13.695 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:13.696 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:13.697 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:13.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:28:13.699 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:13.700 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:13Z" 2024-04-09T22:28:13.702 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:13.705 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:13Z" } 2024-04-09T22:28:13.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:13.706 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:13.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:13.706 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-09T22:28:18.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:18.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:18.708 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:18.759 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:18.759 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:18.803+0000 7f32f94c8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:18.804+0000 7f32f94c8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:18.820 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:28:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:18.827 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:18.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:18.828 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:18Z" 2024-04-09T22:28:18.828 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:18.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:18Z" } 2024-04-09T22:28:18.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:18.829 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:18.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:18.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:28:23.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:23.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:23.831 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:23.882 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:23.882 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:23.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:23.920+0000 7f7c7aabb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:23.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:23.921+0000 7f7c7aabb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:23.935 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.265896_573.1", 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-09T22:28:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:23Z" 2024-04-09T22:28:23.941 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:23.943 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.265896_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.265896Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:23Z" } 2024-04-09T22:28:23.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:23.943 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:23.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:23.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.265896_573.1 2024-04-09T22:28:28.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:28.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:28.944 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:28.996 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:28.996 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:29.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:29.029+0000 7fc436729780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:29.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:29.030+0000 7fc436729780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:29.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:29.047 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:29.048 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:28:29.049 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:29.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:29.051 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:29Z" 2024-04-09T22:28:29.052 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:29.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:29Z" } 2024-04-09T22:28:29.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:29.053 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:29.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:29.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:28:34.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:34.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:34.055 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:34.113 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:34.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:34.176+0000 7f43b7f14780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:34.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:34.184+0000 7f43b7f14780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:28:34.245 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:34.246 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:34Z" 2024-04-09T22:28:34.247 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:34.248 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:34Z" } 2024-04-09T22:28:34.249 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:34.249 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:34.249 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:34.249 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:28:39.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:39.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:39.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:39.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:39.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:39.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:39.383+0000 7fcb18ac3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:39.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:39.384+0000 7fcb18ac3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:39.401 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:39.401 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:39.402 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:39.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:39.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:39.406 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:39Z" 2024-04-09T22:28:39.407 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:39.408 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:39Z" } 2024-04-09T22:28:39.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:39.409 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:39.409 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:39.409 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:28:44.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:44.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:44.410 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:44.461 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:44.461 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:44.513+0000 7fef9b881780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:44.514+0000 7fef9b881780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:44.531 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:44.532 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:44.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:44.534 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.282784_575.1", 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:44.535 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:44.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:44.537 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:44.538 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:44Z" 2024-04-09T22:28:44.538 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:44.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.282784_575.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.282784Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:44Z" } 2024-04-09T22:28:44.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:44.539 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:44.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:44.539 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.282784_575.1 2024-04-09T22:28:49.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:49.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:49.540 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:49.591 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:49.591 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:49.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:49.638+0000 7f8f2931c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:49.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:49.639+0000 7f8f2931c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:49.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:49.661 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:49.662 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.252615_571.1", 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:49.663 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:49.664 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:49Z" 2024-04-09T22:28:49.665 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:49.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.252615_571.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.252615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:49Z" } 2024-04-09T22:28:49.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.252615_571.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.252615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:49.667 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:49.667 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.252615_571.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:49.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.252615_571.1 2024-04-09T22:28:54.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:54.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:54.669 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:54.720 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:54.720 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:54.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:54.761+0000 7fc258588780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:54.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:54.762+0000 7fc258588780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.207085_567.1", 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:54.789 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:54.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:54.790 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:54Z" 2024-04-09T22:28:54.790 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:54.791 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.207085_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.207085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:54Z" } 2024-04-09T22:28:54.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:54.791 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:54.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:54.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.207085_567.1 2024-04-09T22:28:59.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:59.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:28:59.793 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:28:59.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:28:59.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:28:59.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:59.930+0000 7f4c54fdb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:59.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:28:59.931+0000 7f4c54fdb780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:28:59.950 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:28:59.950 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:28:59.950 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:28:59.950 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:28:59.950 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:59.951 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:28:59.952 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:28:59.953 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:28:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:28:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:28:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:28:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:28:59Z" 2024-04-09T22:28:59.956 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:28:59.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:28:59Z" } 2024-04-09T22:28:59.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:28:59.957 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:59.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:28:59.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:29:04.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:04.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:04.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:05.018 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:05.019 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:05.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:05.057+0000 7f9856219780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:05.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:05.058+0000 7f9856219780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:05.075 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:05.075 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:05.076 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:05.077 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:05.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:05.079 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:05.080 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:05Z" 2024-04-09T22:29:05.082 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:05.082 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:05Z" } 2024-04-09T22:29:05.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:05.083 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:05.083 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:05.083 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:29:10.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:10.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:10.085 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:10.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:10.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:10.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:10.221+0000 7f534cdf2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:10.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:10.222+0000 7f534cdf2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:10.237 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:10.237 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:10.237 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:10.237 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:10.238 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:10.239 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:10.240 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.533935_583.1", 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.533935Z", 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:10.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:10.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:10Z" 2024-04-09T22:29:10.243 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:10.244 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.533935_583.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.533935Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:10Z" } 2024-04-09T22:29:10.244 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.533935_583.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.533935Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:10.245 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:10.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.533935_583.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:10.245 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.533935_583.1 2024-04-09T22:29:15.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:15.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:15.247 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:15.298 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:15.299 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:15.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:15.331+0000 7f0771cb8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:15.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:15.332+0000 7f0771cb8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:29:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:15.358 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:15.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:15.358 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:15Z" 2024-04-09T22:29:15.358 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:15.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:15Z" } 2024-04-09T22:29:15.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:15.359 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:15.359 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:15.359 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:29:20.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:20.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:20.361 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:20.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:20.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:20.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:20.471+0000 7f02c81a9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:20.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:20.472+0000 7f02c81a9780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:20.495 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.207085_567.1", 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:20Z" 2024-04-09T22:29:20.501 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:20.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.207085_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.207085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:20Z" } 2024-04-09T22:29:20.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:20.502 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:20.502 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:20.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.207085_567.1 2024-04-09T22:29:25.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:25.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:25.505 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:25.564 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:25.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:25.654+0000 7f7cf1324780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:25.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:25.657+0000 7f7cf1324780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:25.715 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:25.715 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:25.715 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:25.715 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:25.715 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:25.715 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:25.716 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:25.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.197429_564.1", 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:25.718 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:25.719 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:25.720 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:25Z" 2024-04-09T22:29:25.721 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:25.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.197429_564.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.197429Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:25Z" } 2024-04-09T22:29:25.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:25.722 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:25.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:25.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.197429_564.1 2024-04-09T22:29:30.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:30.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:30.723 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:30.773 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:30.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:30.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:30.808+0000 7f819d09c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:30.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:30.809+0000 7f819d09c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:30.823 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:30.824 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:30.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:30.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:30.827 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:30.828 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:30Z" 2024-04-09T22:29:30.829 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:30.830 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:30Z" } 2024-04-09T22:29:30.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:30.831 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:30.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:30.831 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:29:35.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:35.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:35.832 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:35.891 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:35.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:35.932+0000 7ff945ca4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:35.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:35.934+0000 7ff945ca4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:35.950 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:35.951 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:35.952 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:35.953 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:35.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:35.955 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:35Z" 2024-04-09T22:29:35.956 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:35.957 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:35Z" } 2024-04-09T22:29:35.958 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:35.958 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:35.958 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:35.958 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:29:40.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:40.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:40.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:41.016 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:41.017 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:41.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:41.053+0000 7f044e3f2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:41.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:41.055+0000 7f044e3f2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:41.070 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:41.070 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:41.071 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:41.071 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:41.071 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:41.071 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:41.071 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:41.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:41.072 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:41.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:41.072 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:41.072 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:41.072 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:41.073 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:41.073 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:41.073 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:41.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:41.073 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:41.073 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:41.074 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:41.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:41.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:41.074 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:41.074 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:41.074 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:41.075 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:41.075 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:41.075 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:41.075 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:41.076 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:41.076 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:41.076 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:41.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:41.076 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:41.076 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:41.077 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:41.077 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:41.077 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:29:41.077 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:41.077 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:41.078 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:41.079 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:41.080 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:41.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:41.080 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:41Z" 2024-04-09T22:29:41.080 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:41.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:41Z" } 2024-04-09T22:29:41.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:41.081 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:41.081 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:41.081 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:29:46.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:46.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:46.083 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:46.135 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:46.135 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:46.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:46.174+0000 7f0c0eec0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:46.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:46.175+0000 7f0c0eec0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:46.189 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:46.189 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:46.189 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:46.189 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:46.190 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:46.191 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:46.192 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:46.193 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:46.193 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:46.193 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.207085_567.1", 2024-04-09T22:29:46.193 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:46.193 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:46.193 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:46.194 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:46.195 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:46.196 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:46.197 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:46Z" 2024-04-09T22:29:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:46.199 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.207085_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.207085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:46Z" } 2024-04-09T22:29:46.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:46.199 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:46.199 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:46.199 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.207085_567.1 2024-04-09T22:29:51.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:51.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:51.202 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:51.253 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:51.253 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:51.302+0000 7f666853d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:51.303+0000 7f666853d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:51.323 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:51.323 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:51.324 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:51.325 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:51.326 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:51.327 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:51.328 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:51.329 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:51.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:51.329 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:51Z" 2024-04-09T22:29:51.329 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:51.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:51Z" } 2024-04-09T22:29:51.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:51.330 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:51.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:51.330 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:29:56.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:56.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:29:56.332 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:29:56.389 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:29:56.389 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:29:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:56.439+0000 7f66c2ef2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:56.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:29:56.440+0000 7f66c2ef2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:29:56.456 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:29:56.456 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:29:56.456 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:29:56.456 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:29:56.456 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:29:56.456 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:56.457 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:56.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.197429_564.1", 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:56.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:29:56.460 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:29:56Z" 2024-04-09T22:29:56.461 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:29:56.462 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.197429_564.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.197429Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:29:56Z" } 2024-04-09T22:29:56.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:29:56.463 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:56.463 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:29:56.463 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.197429_564.1 2024-04-09T22:30:01.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:01.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:01.464 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:01.516 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:01.516 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:01.557+0000 7f91278a5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:01.558+0000 7f91278a5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:01.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:01.581 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:01.582 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:01.583 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:01.584 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:01Z" 2024-04-09T22:30:01.585 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:01.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:01Z" } 2024-04-09T22:30:01.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:01.587 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:01.587 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:01.587 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-09T22:30:06.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:06.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:06.588 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:06.646 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:06.646 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:06.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:06.703+0000 7f45491c2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:06.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:06.704+0000 7f45491c2780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:06.723 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:06.724 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:06.725 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:06.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:06.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:06.728 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:06Z" 2024-04-09T22:30:06.729 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:06.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:06Z" } 2024-04-09T22:30:06.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:06.731 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:06.731 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:06.731 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:30:11.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:11.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:11.732 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:11.791 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:11.791 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:11.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:11.848+0000 7f28799db780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:11.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:11.849+0000 7f28799db780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:11.868 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:11.869 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:11.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:11.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:11.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:11Z" 2024-04-09T22:30:11.874 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:11.875 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:11Z" } 2024-04-09T22:30:11.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:11.876 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:11.876 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:11.876 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-09T22:30:16.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:16.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:16.877 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:16.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:16.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:16.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:16.990+0000 7f224d342780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:16.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:16.991+0000 7f224d342780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:17.005 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:16Z" 2024-04-09T22:30:17.011 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:17.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:16Z" } 2024-04-09T22:30:17.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:17.012 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:17.012 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:17.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-09T22:30:22.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:22.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:22.014 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:22.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:22.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:22.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:22.114+0000 7f5f12bec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:22.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:22.115+0000 7f5f12bec780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:22.137 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:22.137 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:22.137 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:22.137 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:22.138 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:22.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:22.140 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:22.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:22.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:22Z" 2024-04-09T22:30:22.143 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:22.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:22Z" } 2024-04-09T22:30:22.144 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:22.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:22.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:22.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:30:27.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:27.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:27.146 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:27.201 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:27.202 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:27.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:27.240+0000 7f4a51e3e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:27.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:27.241+0000 7f4a51e3e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:27.260 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:27.260 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:27.260 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:27.261 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:27.262 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:27.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:27.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:27.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:27Z" 2024-04-09T22:30:27.266 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:27.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:27Z" } 2024-04-09T22:30:27.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:27.268 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:27.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:27.268 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:30:32.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:32.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:32.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:32.319 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:32.319 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:32.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:32.354+0000 7f0c06449780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:32.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:32.355+0000 7f0c06449780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:32.371 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.282784_575.1", 2024-04-09T22:30:32.372 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:32.373 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:32.374 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:32.375 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:32.375 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:32.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:32.375 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:32Z" 2024-04-09T22:30:32.375 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:32.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.282784_575.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.282784Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:32Z" } 2024-04-09T22:30:32.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:32.376 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:32.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:32.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.282784_575.1 2024-04-09T22:30:37.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:37.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:37.378 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:37.434 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:37.434 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:37.478+0000 7f4490e6b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:37.479+0000 7f4490e6b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.528865_581.1", 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:37Z" 2024-04-09T22:30:37.508 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:37.509 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.528865_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.528865Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:37Z" } 2024-04-09T22:30:37.509 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.528865_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.528865Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:37.509 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:37.510 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.528865_581.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:37.510 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.528865_581.1 2024-04-09T22:30:42.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:42.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:42.512 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:42.567 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:42.568 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:42.603+0000 7fc5cf562780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:42.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:42.604+0000 7fc5cf562780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:42.631 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:42.631 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:42.631 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:42.631 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:42.632 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:42.632 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:42.633 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:42.634 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:42Z" 2024-04-09T22:30:42.639 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:42.639 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:42Z" } 2024-04-09T22:30:42.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:42.640 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:42.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:42.640 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-09T22:30:47.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:47.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:47.641 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:47.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:47.738+0000 7fbd2aacc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:47.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:47.739+0000 7fbd2aacc780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:47.756 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:47.756 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:47.757 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:47.758 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:30:47.759 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:47.760 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:47.761 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:47.762 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:47Z" 2024-04-09T22:30:47.762 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:47.763 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:47Z" } 2024-04-09T22:30:47.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:47.763 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:47.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:47.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-09T22:30:52.768 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-04-09T22:30:52.769 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-48 2024-04-09T22:30:52.769 DEBUG:boto:path=/xryqqk-48/ 2024-04-09T22:30:52.769 DEBUG:boto:auth_path=/xryqqk-48/ 2024-04-09T22:30:52.769 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.770 DEBUG:boto:Path: /xryqqk-48/ 2024-04-09T22:30:52.770 DEBUG:boto:Data: 2024-04-09T22:30:52.770 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.770 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.770 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.770 DEBUG:boto:Params: {} 2024-04-09T22:30:52.770 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:30:52.771 DEBUG:boto:Token: None 2024-04-09T22:30:52.771 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/ 2024-04-09T22:30:52.771 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e6fThnabiwtLNmTDvuOuX8EpRjY= 2024-04-09T22:30:52.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e6fThnabiwtLNmTDvuOuX8EpRjY=', 'Content-Length': '0'} 2024-04-09T22:30:52.795 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc93aa1992be4ebb-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.795 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-49 2024-04-09T22:30:52.795 DEBUG:boto:path=/xryqqk-49/ 2024-04-09T22:30:52.795 DEBUG:boto:auth_path=/xryqqk-49/ 2024-04-09T22:30:52.795 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.796 DEBUG:boto:Path: /xryqqk-49/ 2024-04-09T22:30:52.796 DEBUG:boto:Data: 2024-04-09T22:30:52.796 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.796 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.796 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.796 DEBUG:boto:Params: {} 2024-04-09T22:30:52.796 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:30:52.796 DEBUG:boto:Token: None 2024-04-09T22:30:52.796 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/ 2024-04-09T22:30:52.797 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/AB5QdnZav1r8ZYK8pYyPEwkCyw= 2024-04-09T22:30:52.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/AB5QdnZav1r8ZYK8pYyPEwkCyw=', 'Content-Length': '0'} 2024-04-09T22:30:52.834 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000041615cde38abbbd4-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.835 DEBUG:boto:path=/xryqqk-48/ 2024-04-09T22:30:52.835 DEBUG:boto:auth_path=/xryqqk-48/ 2024-04-09T22:30:52.835 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.835 DEBUG:boto:Path: /xryqqk-48/ 2024-04-09T22:30:52.835 DEBUG:boto:Data: 2024-04-09T22:30:52.836 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.836 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.836 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.836 DEBUG:boto:Params: {} 2024-04-09T22:30:52.837 DEBUG:boto:Token: None 2024-04-09T22:30:52.837 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/ 2024-04-09T22:30:52.837 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-09T22:30:52.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-09T22:30:52.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef8c29b577dfb6d0-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.840 DEBUG:boto:path=/xryqqk-48/obj1 2024-04-09T22:30:52.840 DEBUG:boto:auth_path=/xryqqk-48/obj1 2024-04-09T22:30:52.840 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.841 DEBUG:boto:Path: /xryqqk-48/obj1 2024-04-09T22:30:52.841 DEBUG:boto:Data: 2024-04-09T22:30:52.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.841 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.841 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.841 DEBUG:boto:Params: {} 2024-04-09T22:30:52.841 DEBUG:boto:Token: None 2024-04-09T22:30:52.842 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/obj1 2024-04-09T22:30:52.842 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:P1DjdPwCEYJlACFoi8tnxLgWz20= 2024-04-09T22:30:52.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:P1DjdPwCEYJlACFoi8tnxLgWz20='} 2024-04-09T22:30:52.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000170feee9d1dd26f2-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.850 DEBUG:boto:path=/xryqqk-48/ 2024-04-09T22:30:52.851 DEBUG:boto:auth_path=/xryqqk-48/ 2024-04-09T22:30:52.851 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.851 DEBUG:boto:Path: /xryqqk-48/ 2024-04-09T22:30:52.851 DEBUG:boto:Data: 2024-04-09T22:30:52.851 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.851 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.851 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.851 DEBUG:boto:Params: {} 2024-04-09T22:30:52.851 DEBUG:boto:Token: None 2024-04-09T22:30:52.852 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/ 2024-04-09T22:30:52.852 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-09T22:30:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-09T22:30:52.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4c4d30098e199f7-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.854 DEBUG:boto:path=/xryqqk-48/obj2 2024-04-09T22:30:52.854 DEBUG:boto:auth_path=/xryqqk-48/obj2 2024-04-09T22:30:52.854 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.854 DEBUG:boto:Path: /xryqqk-48/obj2 2024-04-09T22:30:52.854 DEBUG:boto:Data: 2024-04-09T22:30:52.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.855 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.855 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.855 DEBUG:boto:Params: {} 2024-04-09T22:30:52.855 DEBUG:boto:Token: None 2024-04-09T22:30:52.855 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/obj2 2024-04-09T22:30:52.855 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tEQJB8u0lLSQi9U7D2Y9xQ81RMI= 2024-04-09T22:30:52.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tEQJB8u0lLSQi9U7D2Y9xQ81RMI='} 2024-04-09T22:30:52.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082d28bb78e5d9b46-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.862 DEBUG:boto:path=/xryqqk-48/ 2024-04-09T22:30:52.863 DEBUG:boto:auth_path=/xryqqk-48/ 2024-04-09T22:30:52.863 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.863 DEBUG:boto:Path: /xryqqk-48/ 2024-04-09T22:30:52.863 DEBUG:boto:Data: 2024-04-09T22:30:52.863 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.863 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.863 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.863 DEBUG:boto:Params: {} 2024-04-09T22:30:52.863 DEBUG:boto:Token: None 2024-04-09T22:30:52.864 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/ 2024-04-09T22:30:52.864 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-09T22:30:52.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-09T22:30:52.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000279924474fd57d6f-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.866 DEBUG:boto:path=/xryqqk-48/obj3 2024-04-09T22:30:52.866 DEBUG:boto:auth_path=/xryqqk-48/obj3 2024-04-09T22:30:52.866 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.866 DEBUG:boto:Path: /xryqqk-48/obj3 2024-04-09T22:30:52.866 DEBUG:boto:Data: 2024-04-09T22:30:52.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.867 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.867 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.867 DEBUG:boto:Params: {} 2024-04-09T22:30:52.867 DEBUG:boto:Token: None 2024-04-09T22:30:52.867 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/obj3 2024-04-09T22:30:52.867 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k+Mq83BqJMM77oh0eQQ9VeVMcXk= 2024-04-09T22:30:52.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k+Mq83BqJMM77oh0eQQ9VeVMcXk='} 2024-04-09T22:30:52.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006c85b8d7cc42802-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.911 DEBUG:boto:path=/xryqqk-48/ 2024-04-09T22:30:52.911 DEBUG:boto:auth_path=/xryqqk-48/ 2024-04-09T22:30:52.911 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.911 DEBUG:boto:Path: /xryqqk-48/ 2024-04-09T22:30:52.911 DEBUG:boto:Data: 2024-04-09T22:30:52.911 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.912 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.912 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.912 DEBUG:boto:Params: {} 2024-04-09T22:30:52.912 DEBUG:boto:Token: None 2024-04-09T22:30:52.912 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/ 2024-04-09T22:30:52.912 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-09T22:30:52.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-09T22:30:52.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000204e1169497a67c8-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.915 DEBUG:boto:path=/xryqqk-48/obj4 2024-04-09T22:30:52.915 DEBUG:boto:auth_path=/xryqqk-48/obj4 2024-04-09T22:30:52.915 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.915 DEBUG:boto:Path: /xryqqk-48/obj4 2024-04-09T22:30:52.915 DEBUG:boto:Data: 2024-04-09T22:30:52.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.915 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:30:52.915 DEBUG:boto:Port: 8000 2024-04-09T22:30:52.915 DEBUG:boto:Params: {} 2024-04-09T22:30:52.916 DEBUG:boto:Token: None 2024-04-09T22:30:52.916 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-48/obj4 2024-04-09T22:30:52.916 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:BoQymR+g1pd3wl0ne0wN+t6znAY= 2024-04-09T22:30:52.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BoQymR+g1pd3wl0ne0wN+t6znAY='} 2024-04-09T22:30:52.928 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099f91dcb7c4428cd-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.929 DEBUG:boto:path=/xryqqk-49/ 2024-04-09T22:30:52.929 DEBUG:boto:auth_path=/xryqqk-49/ 2024-04-09T22:30:52.929 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.929 DEBUG:boto:Path: /xryqqk-49/ 2024-04-09T22:30:52.929 DEBUG:boto:Data: 2024-04-09T22:30:52.929 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.929 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.929 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.929 DEBUG:boto:Params: {} 2024-04-09T22:30:52.930 DEBUG:boto:Token: None 2024-04-09T22:30:52.930 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/ 2024-04-09T22:30:52.930 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4= 2024-04-09T22:30:52.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4=', 'Content-Length': '0'} 2024-04-09T22:30:52.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8437be4abeb7ded-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.933 DEBUG:boto:path=/xryqqk-49/obj1 2024-04-09T22:30:52.933 DEBUG:boto:auth_path=/xryqqk-49/obj1 2024-04-09T22:30:52.933 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.933 DEBUG:boto:Path: /xryqqk-49/obj1 2024-04-09T22:30:52.933 DEBUG:boto:Data: 2024-04-09T22:30:52.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.933 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.933 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.934 DEBUG:boto:Params: {} 2024-04-09T22:30:52.934 DEBUG:boto:Token: None 2024-04-09T22:30:52.934 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/obj1 2024-04-09T22:30:52.934 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ivGIpfw03YcGVBTAL+yTN8j9ZnY= 2024-04-09T22:30:52.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ivGIpfw03YcGVBTAL+yTN8j9ZnY='} 2024-04-09T22:30:52.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad4f99a054a248db-006615c19c-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.946 DEBUG:boto:path=/xryqqk-49/ 2024-04-09T22:30:52.946 DEBUG:boto:auth_path=/xryqqk-49/ 2024-04-09T22:30:52.946 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.946 DEBUG:boto:Path: /xryqqk-49/ 2024-04-09T22:30:52.946 DEBUG:boto:Data: 2024-04-09T22:30:52.946 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.946 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.947 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.947 DEBUG:boto:Params: {} 2024-04-09T22:30:52.947 DEBUG:boto:Token: None 2024-04-09T22:30:52.947 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/ 2024-04-09T22:30:52.947 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4= 2024-04-09T22:30:52.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4=', 'Content-Length': '0'} 2024-04-09T22:30:52.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f39161fbcf91ae-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.949 DEBUG:boto:path=/xryqqk-49/obj2 2024-04-09T22:30:52.950 DEBUG:boto:auth_path=/xryqqk-49/obj2 2024-04-09T22:30:52.950 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.950 DEBUG:boto:Path: /xryqqk-49/obj2 2024-04-09T22:30:52.950 DEBUG:boto:Data: 2024-04-09T22:30:52.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.950 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.950 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.950 DEBUG:boto:Params: {} 2024-04-09T22:30:52.950 DEBUG:boto:Token: None 2024-04-09T22:30:52.951 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/obj2 2024-04-09T22:30:52.951 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kbdIFZAoP2Fe14ypIproBHx8aeE= 2024-04-09T22:30:52.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kbdIFZAoP2Fe14ypIproBHx8aeE='} 2024-04-09T22:30:52.971 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b16f48ce3c6f7a62-006615c19c-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.971 DEBUG:boto:path=/xryqqk-49/ 2024-04-09T22:30:52.971 DEBUG:boto:auth_path=/xryqqk-49/ 2024-04-09T22:30:52.971 DEBUG:boto:Method: HEAD 2024-04-09T22:30:52.971 DEBUG:boto:Path: /xryqqk-49/ 2024-04-09T22:30:52.972 DEBUG:boto:Data: 2024-04-09T22:30:52.972 DEBUG:boto:Headers: {} 2024-04-09T22:30:52.972 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.972 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.972 DEBUG:boto:Params: {} 2024-04-09T22:30:52.972 DEBUG:boto:Token: None 2024-04-09T22:30:52.972 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/ 2024-04-09T22:30:52.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4= 2024-04-09T22:30:52.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4=', 'Content-Length': '0'} 2024-04-09T22:30:52.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e97aeb20e3dd9c2a-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:52.982 DEBUG:boto:path=/xryqqk-49/obj3 2024-04-09T22:30:52.983 DEBUG:boto:auth_path=/xryqqk-49/obj3 2024-04-09T22:30:52.983 DEBUG:boto:Method: PUT 2024-04-09T22:30:52.983 DEBUG:boto:Path: /xryqqk-49/obj3 2024-04-09T22:30:52.983 DEBUG:boto:Data: 2024-04-09T22:30:52.983 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:52.983 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:52.983 DEBUG:boto:Port: 8001 2024-04-09T22:30:52.983 DEBUG:boto:Params: {} 2024-04-09T22:30:52.983 DEBUG:boto:Token: None 2024-04-09T22:30:52.983 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:52 GMT /xryqqk-49/obj3 2024-04-09T22:30:52.984 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tFfLzVCn0FwBp1xtxQNsszCjjxk= 2024-04-09T22:30:52.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tFfLzVCn0FwBp1xtxQNsszCjjxk='} 2024-04-09T22:30:53.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000284de68c837f36c0-006615c19c-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:53.027 DEBUG:boto:path=/xryqqk-49/ 2024-04-09T22:30:53.027 DEBUG:boto:auth_path=/xryqqk-49/ 2024-04-09T22:30:53.027 DEBUG:boto:Method: HEAD 2024-04-09T22:30:53.027 DEBUG:boto:Path: /xryqqk-49/ 2024-04-09T22:30:53.027 DEBUG:boto:Data: 2024-04-09T22:30:53.027 DEBUG:boto:Headers: {} 2024-04-09T22:30:53.027 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:53.028 DEBUG:boto:Port: 8001 2024-04-09T22:30:53.028 DEBUG:boto:Params: {} 2024-04-09T22:30:53.028 DEBUG:boto:Token: None 2024-04-09T22:30:53.028 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:30:53 GMT /xryqqk-49/ 2024-04-09T22:30:53.028 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JF3VNm/NdQ2sTI1n0ArNUy7UoOI= 2024-04-09T22:30:53.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JF3VNm/NdQ2sTI1n0ArNUy7UoOI=', 'Content-Length': '0'} 2024-04-09T22:30:53.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008486a311345f2964-006615c19d-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:53.031 DEBUG:boto:path=/xryqqk-49/obj4 2024-04-09T22:30:53.031 DEBUG:boto:auth_path=/xryqqk-49/obj4 2024-04-09T22:30:53.031 DEBUG:boto:Method: PUT 2024-04-09T22:30:53.031 DEBUG:boto:Path: /xryqqk-49/obj4 2024-04-09T22:30:53.031 DEBUG:boto:Data: 2024-04-09T22:30:53.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:30:53.031 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:30:53.032 DEBUG:boto:Port: 8001 2024-04-09T22:30:53.032 DEBUG:boto:Params: {} 2024-04-09T22:30:53.032 DEBUG:boto:Token: None 2024-04-09T22:30:53.032 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:30:53 GMT /xryqqk-49/obj4 2024-04-09T22:30:53.032 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mWDL2SHiURmj6wZpauWVZHqoCTc= 2024-04-09T22:30:53.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mWDL2SHiURmj6wZpauWVZHqoCTc='} 2024-04-09T22:30:53.047 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c75d366053c834bb-006615c19d-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:30:53.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:53.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:53.047 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:53.132 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:53.157 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701852.819902_651.1", 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:30:52.819902Z" 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:53.158 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:30:53.159 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:53.159 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:30:53.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:53.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:53.160 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:53.209 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:53.209 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:53.265+0000 7fc77c6d5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:53.266+0000 7fc77c6d5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:53.287 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:53.288 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:53.289 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701541.570209_607.1", 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:53.290 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:41.570209Z", 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:53.291 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:53.292 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:53Z" 2024-04-09T22:30:53.293 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:53.293 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701541.570209_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:41.570209Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:53Z" } 2024-04-09T22:30:53.294 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.570209_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.570209Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:53.294 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:53.294 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.570209_607.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:53.294 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.570209_607.1 2024-04-09T22:30:58.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:58.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:30:58.295 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:30:58.345 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:30:58.345 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:30:58.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:58.379+0000 7f65c1202780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:58.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:30:58.380+0000 7f65c1202780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:30:58.394 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:30:58.394 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:58.395 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:58.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:58.397 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:30:58.398 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:30:58.399 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:30:58.400 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:30:58.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:30:58.401 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:30:58Z" 2024-04-09T22:30:58.401 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:30:58.402 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:30:58Z" } 2024-04-09T22:30:58.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:30:58.402 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:58.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:30:58.403 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:31:03.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:03.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:03.404 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:03.455 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:03.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:03.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:03.491+0000 7f8147ac0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:03.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:03.492+0000 7f8147ac0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:03.510 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:03.510 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:03.511 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:03.512 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:03.513 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.252615_571.1", 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:03.514 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:03.515 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:03.516 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:03Z" 2024-04-09T22:31:03.516 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:03.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.252615_571.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.252615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:03Z" } 2024-04-09T22:31:03.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.252615_571.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.252615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:03.517 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:03.517 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.252615_571.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:03.518 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.252615_571.1 2024-04-09T22:31:08.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:08.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:08.520 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:08.570 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:08.570 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:08.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:08.619+0000 7f0a0075a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:08.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:08.620+0000 7f0a0075a780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:08.640 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:08.641 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:08.642 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:08.643 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:08.644 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:08Z" 2024-04-09T22:31:08.645 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:08.646 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:08Z" } 2024-04-09T22:31:08.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:08.647 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:08.647 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:08.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-09T22:31:13.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:13.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:13.648 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:13.698 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:13.698 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:13.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:13.733+0000 7fa627d47780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:13.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:13.734+0000 7fa627d47780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:13.748 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:13.749 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:13.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:13.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.270841_574.1", 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:13.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:13.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:13Z" 2024-04-09T22:31:13.754 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:13.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.270841_574.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.270841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:13Z" } 2024-04-09T22:31:13.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.270841_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.270841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:13.756 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:13.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.270841_574.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:13.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.270841_574.1 2024-04-09T22:31:18.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:18.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:18.758 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:18.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:18.863+0000 7f6ef8c5f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:18.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:18.864+0000 7f6ef8c5f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:18.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:18.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:18.883 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.270841_574.1", 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:18.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:18.885 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:18Z" 2024-04-09T22:31:18.886 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:18.888 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.270841_574.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.270841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:18Z" } 2024-04-09T22:31:18.888 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.270841_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.270841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:18.888 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:18.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.270841_574.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:18.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.270841_574.1 2024-04-09T22:31:23.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:23.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:23.890 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:23.981 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:23.981 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:24.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:24.045+0000 7f1e08677780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:24.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:24.046+0000 7f1e08677780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:24.065 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:24.065 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:24.066 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:24.067 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:24.068 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:24.069 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:24.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:24Z" 2024-04-09T22:31:24.071 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:24.072 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:24Z" } 2024-04-09T22:31:24.072 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:24.072 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:24.073 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:24.073 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-09T22:31:29.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:29.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:29.074 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:29.123 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:29.123 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:29.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:29.173+0000 7f2492404780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:29.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:29.174+0000 7f2492404780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:29.188 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:29.188 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:29.188 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:29.188 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:29.188 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:29.189 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:29.190 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.528865_581.1", 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:29.193 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:29.194 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:29.194 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:29.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:29.194 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:29Z" 2024-04-09T22:31:29.194 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:29.195 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.528865_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.528865Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:29Z" } 2024-04-09T22:31:29.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.528865_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.528865Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:29.195 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:29.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.528865_581.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:29.196 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.528865_581.1 2024-04-09T22:31:34.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:34.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:34.197 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:34.254 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:34.255 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:34.312+0000 7fdb130d1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:34.313+0000 7fdb130d1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:34.335 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:34.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:34.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:34.338 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:34.339 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:34.340 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:34.341 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:34Z" 2024-04-09T22:31:34.342 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:34.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:34Z" } 2024-04-09T22:31:34.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:34.343 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:34.343 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:34.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-09T22:31:39.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:39.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:39.345 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:39.397 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:39.398 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:39.433+0000 7f7d1f550780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:39.434+0000 7f7d1f550780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:39.449 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:39.450 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:39.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.282784_575.1", 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:39.453 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:39.454 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:39.455 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:39.455 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:39.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:39.455 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:39Z" 2024-04-09T22:31:39.455 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:39.456 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.282784_575.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.282784Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:39Z" } 2024-04-09T22:31:39.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:39.456 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:39.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:39.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.282784_575.1 2024-04-09T22:31:44.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:44.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:44.458 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:44.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:44.543+0000 7f38b8483780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:44.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:44.544+0000 7f38b8483780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:44.558 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:44.558 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:44.558 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:44.559 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:44.560 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:44.561 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.265896_573.1", 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:44.562 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:44.563 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:44.564 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:44.564 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:44.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:44.564 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:44Z" 2024-04-09T22:31:44.564 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:44.565 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.265896_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.265896Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:44Z" } 2024-04-09T22:31:44.565 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:44.565 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:44.565 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:44.565 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.265896_573.1 2024-04-09T22:31:49.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:49.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:49.567 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:49.623 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:49.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:49.660+0000 7f8912ded780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:49.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:49.661+0000 7f8912ded780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:49.677 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:49.678 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:49.679 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.207085_567.1", 2024-04-09T22:31:49.680 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:49Z" 2024-04-09T22:31:49.683 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:49.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.207085_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.207085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:49Z" } 2024-04-09T22:31:49.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:49.685 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:49.685 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:49.685 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.207085_567.1 2024-04-09T22:31:54.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:54.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:54.686 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:54.746 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:54.746 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:54.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:54.805+0000 7f366cb5d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:54.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:54.806+0000 7f366cb5d780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:54.819 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:54.819 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:54.819 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:54.820 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:54.821 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:54.822 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:54.823 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.265896_573.1", 2024-04-09T22:31:54.823 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:54.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:54.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:54.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-09T22:31:54.823 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:54.824 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:54Z" 2024-04-09T22:31:54.825 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:54.826 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.265896_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.265896Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:54Z" } 2024-04-09T22:31:54.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:54.827 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:54.827 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:54.827 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.265896_573.1 2024-04-09T22:31:59.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:59.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:31:59.829 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:31:59.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:31:59.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:31:59.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:59.939+0000 7fd32a9d4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:59.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:31:59.940+0000 7fd32a9d4780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:31:59.954 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:31:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:31:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:31:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:31:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:31:59.954 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:31:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:59.956 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:59.957 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701541.011327_591.1", 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:59.958 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:41.011327Z", 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:31:59.959 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:31:59.960 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:31:59Z" 2024-04-09T22:31:59.961 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:31:59.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701541.011327_591.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:41.011327Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:31:59Z" } 2024-04-09T22:31:59.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.011327_591.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.011327Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:31:59.963 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:59.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.011327_591.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:31:59.963 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.011327_591.1 2024-04-09T22:32:04.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:04.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:04.965 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:05.022 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:05.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:05.063+0000 7f75701ab780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:05.064+0000 7f75701ab780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:05.080 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:05.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:05.082 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:05.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:05.084 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:05.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:05.086 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:05.087 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:05.088 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:05.089 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:05.090 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:05Z" 2024-04-09T22:32:05.090 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:05.091 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:05Z" } 2024-04-09T22:32:05.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:05.091 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:05.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:05.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-09T22:32:10.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:10.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:10.093 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:10.150 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:10.150 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:10.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:10.207+0000 7f2b3574b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:10.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:10.208+0000 7f2b3574b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:10.229 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:10.230 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:10.231 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701541.430664_598.1", 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:10.232 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:41.430664Z", 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:10.233 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:10.234 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:10Z" 2024-04-09T22:32:10.235 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:10.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701541.430664_598.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:41.430664Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:10Z" } 2024-04-09T22:32:10.236 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.430664_598.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.430664Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:10.236 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:10.236 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.430664_598.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:10.236 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.430664_598.1 2024-04-09T22:32:12.317 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:32:12.317+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:32:15.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:15.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:15.238 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:15.289 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:15.289 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:15.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:15.331+0000 7fddf0202780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:15.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:15.332+0000 7fddf0202780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:15.350 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:15.351 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:15.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:15.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.265896_573.1", 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:15.354 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-09T22:32:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:15.355 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:15.356 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:15.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:15.358 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:15.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:15.358 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:15Z" 2024-04-09T22:32:15.358 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:15.359 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.265896_573.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.265896Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:15Z" } 2024-04-09T22:32:15.359 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:15.360 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:15.360 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:15.360 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.265896_573.1 2024-04-09T22:32:20.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:20.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:20.362 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:20.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:20.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:20.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:20.459+0000 7f0900d53780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:20.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:20.460+0000 7f0900d53780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:20.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:20.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:20.478 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701543.601335_611.1", 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:20.479 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:43.601335Z", 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:20.480 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:20.481 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:20.481 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:20.481 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:20.482 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:20Z" 2024-04-09T22:32:20.483 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:20.483 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701543.601335_611.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:43.601335Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:20Z" } 2024-04-09T22:32:20.484 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701543.601335_611.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:43.601335Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:20.484 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:20.484 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701543.601335_611.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:20.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701543.601335_611.1 2024-04-09T22:32:25.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:25.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:25.485 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:25.542 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:25.543 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:25.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:25.579+0000 7ffaed00b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:25.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:25.580+0000 7ffaed00b780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:25.598 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:25.598 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:25.599 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:25.600 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:25.601 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701541.012414_592.1", 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:41.012414Z", 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:25.602 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:25.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:25.604 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:25Z" 2024-04-09T22:32:25.604 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:25.605 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701541.012414_592.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:41.012414Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:25Z" } 2024-04-09T22:32:25.605 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.012414_592.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.012414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:25.605 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:25.605 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.012414_592.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:25.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.012414_592.1 2024-04-09T22:32:30.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:30.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:30.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:30.660 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:30.660 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:30.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:30.713+0000 7fa83c982780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:30.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:30.714+0000 7fa83c982780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:30.731 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:30.732 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:30.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:30.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.197429_564.1", 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:30.735 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:30.736 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:30.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:30.738 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:30Z" 2024-04-09T22:32:30.738 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:30.739 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.197429_564.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.197429Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:30Z" } 2024-04-09T22:32:30.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:30.739 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:30.739 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:30.739 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-09T22:32:35.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:35.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:35.740 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:35.793 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:35.793 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:35.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:35.826+0000 7f4b904ed780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:35.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:35.827+0000 7f4b904ed780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:35.842 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:35Z" 2024-04-09T22:32:35.847 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:35.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:35Z" } 2024-04-09T22:32:35.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:35.848 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:35.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:35.849 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-09T22:32:40.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:40.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:40.850 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:40.910 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:40.946+0000 7fb34d280780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:40.947+0000 7fb34d280780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:40.966 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.207085_567.1", 2024-04-09T22:32:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:40Z" 2024-04-09T22:32:40.973 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:40.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.207085_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.207085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:40Z" } 2024-04-09T22:32:40.975 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:40.975 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:40.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:40.975 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.207085_567.1 2024-04-09T22:32:45.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:45.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:45.977 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:46.028 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:46.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:46.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:46.083+0000 7f61ea2e5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:46.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:46.085+0000 7f61ea2e5780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:46.114 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:46.114 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:46Z" 2024-04-09T22:32:46.120 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:46.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:46Z" } 2024-04-09T22:32:46.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:46.121 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:46.121 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:46.121 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:32:51.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:51.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:51.123 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:51.179 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:51.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:51.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:51.224+0000 7f6b0a6b7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:51.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:51.225+0000 7f6b0a6b7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:51.244 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:51.244 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:51.244 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:51.244 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:51.244 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:51.245 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:51.246 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:51.247 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:51.248 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:51Z" 2024-04-09T22:32:51.250 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:51.251 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:51Z" } 2024-04-09T22:32:51.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:51.251 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:51.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:51.251 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-09T22:32:56.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:56.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:32:56.252 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:32:56.311 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:32:56.311 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:32:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:56.352+0000 7f44d0a18780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:32:56.353+0000 7f44d0a18780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:32:56.375 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:56.376 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:56.377 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:56.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:32:56.379 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:32:56.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:56.381 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:32:56.381 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:32:56.381 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:32:56.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:32:56.381 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:32:56Z" 2024-04-09T22:32:56.381 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:32:56.382 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:32:56Z" } 2024-04-09T22:32:56.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:32:56.382 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:56.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:32:56.383 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:33:01.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:01.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:01.384 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:01.434 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:01.434 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:01.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:01.470+0000 7f7f46667780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:01.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:01.471+0000 7f7f46667780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:01.489 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:01.489 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:01.490 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:01.491 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:01.492 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.197429_564.1", 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:01.493 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:01.494 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:01Z" 2024-04-09T22:33:01.495 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:01.496 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.197429_564.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.197429Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:01Z" } 2024-04-09T22:33:01.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:01.496 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:01.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:01.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-09T22:33:06.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:06.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:06.498 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:06.558 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:06.558 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:06.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:06.610+0000 7f25cfbc1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:06.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:06.611+0000 7f25cfbc1780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:06.626 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:06.626 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:06.627 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:06.628 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:06.629 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:06.630 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:06.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:06.632 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:06Z" 2024-04-09T22:33:06.632 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:06.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:06Z" } 2024-04-09T22:33:06.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:06.633 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:06.633 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:06.633 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:33:11.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:11.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:11.635 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:11.691 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:11.691 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:11.728+0000 7f69f6b8c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:11.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:11.729+0000 7f69f6b8c780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:11.744 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:11.744 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:11.745 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:11.746 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:11.747 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.282784_575.1", 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:11.748 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:11.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:11.750 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:11.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:11.750 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:11Z" 2024-04-09T22:33:11.750 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:11.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.282784_575.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.282784Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:11Z" } 2024-04-09T22:33:11.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:11.751 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:11.751 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:11.751 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.282784_575.1 2024-04-09T22:33:16.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:16.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:16.753 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:16.804 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:16.805 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:16.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:16.860+0000 7f312e852780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:16.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:16.861+0000 7f312e852780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.207085_567.1", 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:16Z" 2024-04-09T22:33:16.889 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:16.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.207085_567.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.207085Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:16Z" } 2024-04-09T22:33:16.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:16.890 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:16.890 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:16.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.207085_567.1 2024-04-09T22:33:21.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:21.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:21.892 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:21.947 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:21.947 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:22.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:22.001+0000 7f931a97f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:22.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:22.002+0000 7f931a97f780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:22.025 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:22.026 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:22.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:22.028 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:22.028 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:22.028 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:22.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:22.028 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:22.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:33:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:22.031 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:22.031 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:33:22.031 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:22.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:22.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:22.032 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:22.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:22.034 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:22Z" 2024-04-09T22:33:22.034 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:22.035 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:22Z" } 2024-04-09T22:33:22.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:22.035 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:22.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:22.035 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:33:27.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:27.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:27.037 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:27.121 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:27.121 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:27.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:27.202+0000 7f8cf5bf0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:27.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:27.203+0000 7f8cf5bf0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:27.227 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:27.228 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:27.229 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:27Z" 2024-04-09T22:33:27.230 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:27.231 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:27Z" } 2024-04-09T22:33:27.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:27.232 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:27.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:27.232 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:33:32.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:32.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:32.234 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:32.293 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:32.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:32.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:32.341+0000 7f27829a8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:32.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:32.342+0000 7f27829a8780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:32.363 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701541.012414_592.1", 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:32.367 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:41.012414Z", 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:32.368 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:32.369 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:32Z" 2024-04-09T22:33:32.370 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:32.372 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701541.012414_592.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:41.012414Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:32Z" } 2024-04-09T22:33:32.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.012414_592.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.012414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:32.373 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:32.373 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.012414_592.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:32.373 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.012414_592.1 2024-04-09T22:33:37.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:37.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:37.375 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:37.433 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:37.485+0000 7f98609b6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:37.486+0000 7f98609b6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:37.503 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:37.506 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.528865_581.1", 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:37.507 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:37.508 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:37.509 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:37Z" 2024-04-09T22:33:37.510 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:37.510 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.528865_581.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.528865Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:37Z" } 2024-04-09T22:33:37.511 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.528865_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.528865Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:37.511 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:37.511 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.528865_581.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:37.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.528865_581.1 2024-04-09T22:33:42.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:42.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:42.513 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:42.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:42.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:42.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:42.662+0000 7fac2c8d3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:42.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:42.663+0000 7fac2c8d3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:42.683 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:42.684 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:42.685 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:42.686 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701541.008529_590.1", 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:42.687 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:41.008529Z", 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:42.688 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:42Z" 2024-04-09T22:33:42.689 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:42.691 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701541.008529_590.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:41.008529Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:42Z" } 2024-04-09T22:33:42.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.008529_590.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.008529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:42.691 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:42.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.008529_590.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:42.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.008529_590.1 2024-04-09T22:33:47.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:47.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:47.692 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:47.794+0000 7f16abd40780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:47.795+0000 7f16abd40780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:47.812 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:47.812 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:47.812 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:47.812 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:47.813 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:47.814 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:47.815 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.252615_571.1", 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:47.817 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:47.818 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:47.819 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:47Z" 2024-04-09T22:33:47.819 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:47.820 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.252615_571.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.252615Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:47Z" } 2024-04-09T22:33:47.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.252615_571.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.252615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:47.820 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:47.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.252615_571.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:47.820 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.252615_571.1 2024-04-09T22:33:52.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:52.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:52.822 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:52.873 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:52.873 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:52.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:52.916+0000 7f0a167f6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:52.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:52.917+0000 7f0a167f6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:52.932 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:52.932 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:52.932 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:52.932 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:52.933 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:52.934 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:52.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.270841_574.1", 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-09T22:33:52.936 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:52.937 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:52Z" 2024-04-09T22:33:52.938 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:52.939 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.270841_574.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.270841Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:52Z" } 2024-04-09T22:33:52.940 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.270841_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.270841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:52.940 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:52.940 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.270841_574.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:52.940 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.270841_574.1 2024-04-09T22:33:57.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:57.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:33:57.941 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:33:58.004 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:33:58.004 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:33:58.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:58.063+0000 7f14ddc6e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:58.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:33:58.064+0000 7f14ddc6e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:33:58.085 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:33:58.085 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:33:58.085 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:33:58.085 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:33:58.085 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:33:58.086 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:58.087 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:58.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.197429_564.1", 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:58.089 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:33:58.090 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:58.091 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:33:58.092 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:33:58.092 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:33:58.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:33:58.092 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:33:58Z" 2024-04-09T22:33:58.092 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:33:58.093 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.197429_564.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.197429Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:33:58Z" } 2024-04-09T22:33:58.093 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:33:58.093 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:58.093 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:33:58.093 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-09T22:34:03.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:03.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:03.095 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:03.152 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:03.152 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:03.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:03.196+0000 7efee1756780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:03.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:03.197+0000 7efee1756780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:03.212 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:03.213 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:03.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:03.216 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:03.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:03Z" 2024-04-09T22:34:03.218 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:03.219 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:03Z" } 2024-04-09T22:34:03.219 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:03.220 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:03.220 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:03.220 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-09T22:34:08.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:08.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:08.222 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:08.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:08.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:08.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:08.330+0000 7f221e9dd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:08.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:08.331+0000 7f221e9dd780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:08.385 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:08.385 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:08.386 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:08.387 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:08.388 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.197429_564.1", 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:08.389 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:08.390 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:08Z" 2024-04-09T22:34:08.391 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:08.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.197429_564.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.197429Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:08Z" } 2024-04-09T22:34:08.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:08.392 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:08.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:08.393 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-09T22:34:13.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:13.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:13.394 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:13.445 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:13.445 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:13.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:13.478+0000 7f49ba6c7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:13.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:13.479+0000 7f49ba6c7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:13.493 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:13.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:13.495 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.202102_565.1", 2024-04-09T22:34:13.496 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:13.499 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:13.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:13.499 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:13Z" 2024-04-09T22:34:13.499 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:13.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.202102_565.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.202102Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:13Z" } 2024-04-09T22:34:13.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:13.500 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:13.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:13.500 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-09T22:34:18.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:18.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:18.502 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:18.555 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:18.555 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:18.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:18.592+0000 7fd5e57a3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:18.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:18.593+0000 7fd5e57a3780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:18.608 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:18.609 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:18.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:18.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:18.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:18.612 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:18.612 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:18.614 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:18.614 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:18.614 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:18.615 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:18.616 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:18.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:18.616 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:18Z" 2024-04-09T22:34:18.616 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:18.618 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:18Z" } 2024-04-09T22:34:18.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:18.618 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:18.618 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:18.618 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-09T22:34:23.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:23.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:23.620 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:23.671 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:23.672 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:23.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:23.722+0000 7f54ff5e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:23.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:23.723+0000 7f54ff5e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:23.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:23.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.530215_582.1", 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.530215Z", 2024-04-09T22:34:23.750 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:23.751 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:23Z" 2024-04-09T22:34:23.752 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:23.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.530215_582.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.530215Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:23Z" } 2024-04-09T22:34:23.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.530215_582.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.530215Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:23.753 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:23.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.530215_582.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:23.753 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.530215_582.1 2024-04-09T22:34:28.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:28.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:28.755 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:28.809 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:28.809 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:28.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:28.853+0000 7fbada41e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:28.854+0000 7fbada41e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:28.869 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:28.869 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:28.869 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:28.869 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:28.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:28.871 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:28.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.524385_580.1", 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:28.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:28.874 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:28.875 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:28Z" 2024-04-09T22:34:28.876 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:28.877 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.524385_580.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.524385Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:28Z" } 2024-04-09T22:34:28.877 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:28.878 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:28.878 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:28.878 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-09T22:34:33.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:33.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:33.879 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:33.937 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:33.937 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:33.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:33.981+0000 7f0673417780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:33.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:33.983+0000 7f0673417780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:33.998 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:33.998 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:33.998 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:33.999 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:34.000 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:34.001 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701540.902795_586.1", 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:34.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:34.003 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:34.004 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:33Z" 2024-04-09T22:34:34.005 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:34.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701540.902795_586.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:25:40.902795Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:33Z" } 2024-04-09T22:34:34.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:34.006 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:34.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:34.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-09T22:34:39.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:39.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:34:39.008 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:34:39.060 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:39.060 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:39.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:39.111+0000 7fb634d18780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:39.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:39.112+0000 7fb634d18780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:34:39.134 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:34:39.135 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:39.136 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:39.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712701852.819902_651.1", 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:30:52.819902Z", 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:39.138 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:34:39.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:34:39Z" 2024-04-09T22:34:39.140 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:34:39.141 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712701852.819902_651.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:30:52.819902Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:34:39Z" } 2024-04-09T22:34:39.141 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701852.819902_651.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:30:52.819902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:34:39.142 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:39.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:34:39.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:34:39.142 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-48 2024-04-09T22:34:39.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:34:39.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:34:39.142 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:34:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:39.251 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-48 2024-04-09T22:34:39.271 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-48 2024-04-09T22:34:39.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:34:39.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:34:39.272 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:34:39.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:34:39.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:34:39.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:39.368+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:34:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:44.369+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:34:49.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:49.369+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:34:54.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:54.370+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:34:59.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:34:59.370+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:04.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:04.371+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:09.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:09.371+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:14.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:14.372+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:19.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:19.373+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:24.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:24.374+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:29.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:29.374+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:34.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:34.374+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:39.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:39.375+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:44.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:44.375+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:49.376+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:54.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:54.376+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:35:59.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:35:59.377+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:04.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:04.377+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:09.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:09.378+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:14.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:14.379+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:19.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:19.379+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:24.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:24.380+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:29.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:29.380+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:34.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:34.381+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:39.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:39.381+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:44.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:44.382+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:49.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:49.382+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:54.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:54.383+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:36:59.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:36:59.383+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:04.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:04.385+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:09.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:09.386+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:12.320 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:37:12.320+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:37:14.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:14.385+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:19.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:19.386+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:24.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:24.386+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:29.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:29.387+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:34.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:34.388+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:39.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:39.388+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:44.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:44.389+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:49.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:49.389+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:54.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:54.390+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:37:59.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:37:59.391+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:04.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:04.391+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:09.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:09.392+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:14.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:14.393+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:19.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:19.393+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:24.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:24.394+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:29.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:29.394+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:34.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:34.395+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:39.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:39.396+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:44.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:44.396+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:49.397+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:54.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:54.398+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:38:59.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:38:59.397+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:04.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:04.398+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:09.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:09.398+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:14.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:14.399+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:19.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:19.400+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:24.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:24.400+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:29.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:29.401+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-09T22:39:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:34.402+0000 7fe5a5201780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T22:39:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:34.402+0000 7fe5a5201780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:39:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:34.402+0000 7fe5a5201780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:39:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T22:39:34.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-04-09T22:39:34.426 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-50 2024-04-09T22:39:34.426 DEBUG:boto:path=/xryqqk-50/ 2024-04-09T22:39:34.427 DEBUG:boto:auth_path=/xryqqk-50/ 2024-04-09T22:39:34.427 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.427 DEBUG:boto:Path: /xryqqk-50/ 2024-04-09T22:39:34.427 DEBUG:boto:Data: 2024-04-09T22:39:34.427 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.427 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.427 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.427 DEBUG:boto:Params: {} 2024-04-09T22:39:34.428 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:39:34.428 DEBUG:boto:Token: None 2024-04-09T22:39:34.428 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/ 2024-04-09T22:39:34.428 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qLg2V09L/CpBoOqDvoNHgxsW0Ow= 2024-04-09T22:39:34.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qLg2V09L/CpBoOqDvoNHgxsW0Ow=', 'Content-Length': '0'} 2024-04-09T22:39:34.454 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b8f8d04aca85619a-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.454 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-51 2024-04-09T22:39:34.454 DEBUG:boto:path=/xryqqk-51/ 2024-04-09T22:39:34.454 DEBUG:boto:auth_path=/xryqqk-51/ 2024-04-09T22:39:34.454 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.454 DEBUG:boto:Path: /xryqqk-51/ 2024-04-09T22:39:34.454 DEBUG:boto:Data: 2024-04-09T22:39:34.454 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.454 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.455 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.455 DEBUG:boto:Params: {} 2024-04-09T22:39:34.455 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:39:34.455 DEBUG:boto:Token: None 2024-04-09T22:39:34.455 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/ 2024-04-09T22:39:34.455 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aat8SuVMcnyF3RBB8x+QcjdaBs8= 2024-04-09T22:39:34.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aat8SuVMcnyF3RBB8x+QcjdaBs8=', 'Content-Length': '0'} 2024-04-09T22:39:34.492 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1542495d10f1f09-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.493 DEBUG:boto:path=/xryqqk-50/ 2024-04-09T22:39:34.493 DEBUG:boto:auth_path=/xryqqk-50/ 2024-04-09T22:39:34.493 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.493 DEBUG:boto:Path: /xryqqk-50/ 2024-04-09T22:39:34.493 DEBUG:boto:Data: 2024-04-09T22:39:34.493 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.493 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.493 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.493 DEBUG:boto:Params: {} 2024-04-09T22:39:34.493 DEBUG:boto:Token: None 2024-04-09T22:39:34.494 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/ 2024-04-09T22:39:34.494 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-09T22:39:34.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-09T22:39:34.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014dd47594379c36c-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.496 DEBUG:boto:path=/xryqqk-50/obj1 2024-04-09T22:39:34.497 DEBUG:boto:auth_path=/xryqqk-50/obj1 2024-04-09T22:39:34.497 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.497 DEBUG:boto:Path: /xryqqk-50/obj1 2024-04-09T22:39:34.497 DEBUG:boto:Data: 2024-04-09T22:39:34.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.497 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.497 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.497 DEBUG:boto:Params: {} 2024-04-09T22:39:34.497 DEBUG:boto:Token: None 2024-04-09T22:39:34.497 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/obj1 2024-04-09T22:39:34.497 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6MdLtm6kX8l+pAHlRu59mJIkbUA= 2024-04-09T22:39:34.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6MdLtm6kX8l+pAHlRu59mJIkbUA='} 2024-04-09T22:39:34.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2c808c5aa52f1df-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.505 DEBUG:boto:path=/xryqqk-50/ 2024-04-09T22:39:34.505 DEBUG:boto:auth_path=/xryqqk-50/ 2024-04-09T22:39:34.505 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.505 DEBUG:boto:Path: /xryqqk-50/ 2024-04-09T22:39:34.506 DEBUG:boto:Data: 2024-04-09T22:39:34.506 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.506 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.506 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.506 DEBUG:boto:Params: {} 2024-04-09T22:39:34.506 DEBUG:boto:Token: None 2024-04-09T22:39:34.506 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/ 2024-04-09T22:39:34.506 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-09T22:39:34.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-09T22:39:34.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000295e113317febf78-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.509 DEBUG:boto:path=/xryqqk-50/obj2 2024-04-09T22:39:34.509 DEBUG:boto:auth_path=/xryqqk-50/obj2 2024-04-09T22:39:34.509 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.509 DEBUG:boto:Path: /xryqqk-50/obj2 2024-04-09T22:39:34.509 DEBUG:boto:Data: 2024-04-09T22:39:34.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.509 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.509 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.510 DEBUG:boto:Params: {} 2024-04-09T22:39:34.510 DEBUG:boto:Token: None 2024-04-09T22:39:34.510 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/obj2 2024-04-09T22:39:34.510 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SRoamgSHg3ApEoidjYp3o4yjQTA= 2024-04-09T22:39:34.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SRoamgSHg3ApEoidjYp3o4yjQTA='} 2024-04-09T22:39:34.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d42a2d2ee965fd6-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.516 DEBUG:boto:path=/xryqqk-50/ 2024-04-09T22:39:34.516 DEBUG:boto:auth_path=/xryqqk-50/ 2024-04-09T22:39:34.516 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.516 DEBUG:boto:Path: /xryqqk-50/ 2024-04-09T22:39:34.516 DEBUG:boto:Data: 2024-04-09T22:39:34.517 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.517 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.517 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.517 DEBUG:boto:Params: {} 2024-04-09T22:39:34.517 DEBUG:boto:Token: None 2024-04-09T22:39:34.517 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/ 2024-04-09T22:39:34.517 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-09T22:39:34.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-09T22:39:34.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae367eb33e98f8a0-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.519 DEBUG:boto:path=/xryqqk-50/obj3 2024-04-09T22:39:34.519 DEBUG:boto:auth_path=/xryqqk-50/obj3 2024-04-09T22:39:34.520 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.520 DEBUG:boto:Path: /xryqqk-50/obj3 2024-04-09T22:39:34.520 DEBUG:boto:Data: 2024-04-09T22:39:34.520 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.520 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.520 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.520 DEBUG:boto:Params: {} 2024-04-09T22:39:34.520 DEBUG:boto:Token: None 2024-04-09T22:39:34.520 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/obj3 2024-04-09T22:39:34.520 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6FYHZNZ68U1RgjxCIdHYikQbMx0= 2024-04-09T22:39:34.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6FYHZNZ68U1RgjxCIdHYikQbMx0='} 2024-04-09T22:39:34.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000577fa1a71a9e8f9e-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.567 DEBUG:boto:path=/xryqqk-50/ 2024-04-09T22:39:34.567 DEBUG:boto:auth_path=/xryqqk-50/ 2024-04-09T22:39:34.567 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.567 DEBUG:boto:Path: /xryqqk-50/ 2024-04-09T22:39:34.567 DEBUG:boto:Data: 2024-04-09T22:39:34.567 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.567 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.568 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.568 DEBUG:boto:Params: {} 2024-04-09T22:39:34.568 DEBUG:boto:Token: None 2024-04-09T22:39:34.568 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/ 2024-04-09T22:39:34.568 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-09T22:39:34.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-09T22:39:34.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a87e8d2976d50cc-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.571 DEBUG:boto:path=/xryqqk-50/obj4 2024-04-09T22:39:34.571 DEBUG:boto:auth_path=/xryqqk-50/obj4 2024-04-09T22:39:34.571 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.571 DEBUG:boto:Path: /xryqqk-50/obj4 2024-04-09T22:39:34.571 DEBUG:boto:Data: 2024-04-09T22:39:34.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.571 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:39:34.571 DEBUG:boto:Port: 8000 2024-04-09T22:39:34.571 DEBUG:boto:Params: {} 2024-04-09T22:39:34.571 DEBUG:boto:Token: None 2024-04-09T22:39:34.572 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-50/obj4 2024-04-09T22:39:34.572 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MlepQsXi+A3qSyIkeO1WvpdbEuU= 2024-04-09T22:39:34.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MlepQsXi+A3qSyIkeO1WvpdbEuU='} 2024-04-09T22:39:34.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004294a534cf82e990-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.581 DEBUG:boto:path=/xryqqk-51/ 2024-04-09T22:39:34.581 DEBUG:boto:auth_path=/xryqqk-51/ 2024-04-09T22:39:34.581 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.582 DEBUG:boto:Path: /xryqqk-51/ 2024-04-09T22:39:34.582 DEBUG:boto:Data: 2024-04-09T22:39:34.582 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.582 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.582 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.582 DEBUG:boto:Params: {} 2024-04-09T22:39:34.582 DEBUG:boto:Token: None 2024-04-09T22:39:34.582 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/ 2024-04-09T22:39:34.582 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-09T22:39:34.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-09T22:39:34.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f6c15b5045b9fda-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.585 DEBUG:boto:path=/xryqqk-51/obj1 2024-04-09T22:39:34.585 DEBUG:boto:auth_path=/xryqqk-51/obj1 2024-04-09T22:39:34.586 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.586 DEBUG:boto:Path: /xryqqk-51/obj1 2024-04-09T22:39:34.586 DEBUG:boto:Data: 2024-04-09T22:39:34.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.586 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.586 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.586 DEBUG:boto:Params: {} 2024-04-09T22:39:34.586 DEBUG:boto:Token: None 2024-04-09T22:39:34.586 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/obj1 2024-04-09T22:39:34.587 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CBDc+h/znwMESDV1oGTGMTuKm98= 2024-04-09T22:39:34.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CBDc+h/znwMESDV1oGTGMTuKm98='} 2024-04-09T22:39:34.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8972415332b3fd7-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.593 DEBUG:boto:path=/xryqqk-51/ 2024-04-09T22:39:34.593 DEBUG:boto:auth_path=/xryqqk-51/ 2024-04-09T22:39:34.594 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.594 DEBUG:boto:Path: /xryqqk-51/ 2024-04-09T22:39:34.594 DEBUG:boto:Data: 2024-04-09T22:39:34.594 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.594 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.594 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.594 DEBUG:boto:Params: {} 2024-04-09T22:39:34.594 DEBUG:boto:Token: None 2024-04-09T22:39:34.595 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/ 2024-04-09T22:39:34.595 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-09T22:39:34.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-09T22:39:34.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008b6ceb0909c5b1b-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.597 DEBUG:boto:path=/xryqqk-51/obj2 2024-04-09T22:39:34.597 DEBUG:boto:auth_path=/xryqqk-51/obj2 2024-04-09T22:39:34.597 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.597 DEBUG:boto:Path: /xryqqk-51/obj2 2024-04-09T22:39:34.597 DEBUG:boto:Data: 2024-04-09T22:39:34.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.597 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.597 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.598 DEBUG:boto:Params: {} 2024-04-09T22:39:34.598 DEBUG:boto:Token: None 2024-04-09T22:39:34.598 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/obj2 2024-04-09T22:39:34.598 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Dxqb22Cfcr3OnszFBd//4JTGOHg= 2024-04-09T22:39:34.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Dxqb22Cfcr3OnszFBd//4JTGOHg='} 2024-04-09T22:39:34.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a961fca81ab338c-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.604 DEBUG:boto:path=/xryqqk-51/ 2024-04-09T22:39:34.604 DEBUG:boto:auth_path=/xryqqk-51/ 2024-04-09T22:39:34.604 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.604 DEBUG:boto:Path: /xryqqk-51/ 2024-04-09T22:39:34.604 DEBUG:boto:Data: 2024-04-09T22:39:34.604 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.605 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.605 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.605 DEBUG:boto:Params: {} 2024-04-09T22:39:34.605 DEBUG:boto:Token: None 2024-04-09T22:39:34.605 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/ 2024-04-09T22:39:34.605 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-09T22:39:34.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-09T22:39:34.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075ac6d8fbfea1c92-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.608 DEBUG:boto:path=/xryqqk-51/obj3 2024-04-09T22:39:34.608 DEBUG:boto:auth_path=/xryqqk-51/obj3 2024-04-09T22:39:34.608 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.608 DEBUG:boto:Path: /xryqqk-51/obj3 2024-04-09T22:39:34.608 DEBUG:boto:Data: 2024-04-09T22:39:34.608 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.608 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.608 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.608 DEBUG:boto:Params: {} 2024-04-09T22:39:34.608 DEBUG:boto:Token: None 2024-04-09T22:39:34.609 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/obj3 2024-04-09T22:39:34.609 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EsyQmDPQTCksxSIJbr2LAQ07r98= 2024-04-09T22:39:34.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EsyQmDPQTCksxSIJbr2LAQ07r98='} 2024-04-09T22:39:34.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c693319638ef7388-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.651 DEBUG:boto:path=/xryqqk-51/ 2024-04-09T22:39:34.651 DEBUG:boto:auth_path=/xryqqk-51/ 2024-04-09T22:39:34.651 DEBUG:boto:Method: HEAD 2024-04-09T22:39:34.651 DEBUG:boto:Path: /xryqqk-51/ 2024-04-09T22:39:34.651 DEBUG:boto:Data: 2024-04-09T22:39:34.651 DEBUG:boto:Headers: {} 2024-04-09T22:39:34.651 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.651 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.651 DEBUG:boto:Params: {} 2024-04-09T22:39:34.651 DEBUG:boto:Token: None 2024-04-09T22:39:34.652 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/ 2024-04-09T22:39:34.652 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-09T22:39:34.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-09T22:39:34.654 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a659987deefb882-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.654 DEBUG:boto:path=/xryqqk-51/obj4 2024-04-09T22:39:34.654 DEBUG:boto:auth_path=/xryqqk-51/obj4 2024-04-09T22:39:34.655 DEBUG:boto:Method: PUT 2024-04-09T22:39:34.655 DEBUG:boto:Path: /xryqqk-51/obj4 2024-04-09T22:39:34.655 DEBUG:boto:Data: 2024-04-09T22:39:34.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T22:39:34.655 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:39:34.655 DEBUG:boto:Port: 8001 2024-04-09T22:39:34.655 DEBUG:boto:Params: {} 2024-04-09T22:39:34.655 DEBUG:boto:Token: None 2024-04-09T22:39:34.656 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 22:39:34 GMT /xryqqk-51/obj4 2024-04-09T22:39:34.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:E+a60A2n/maFBNnPkSe5ejj5u9w= 2024-04-09T22:39:34.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E+a60A2n/maFBNnPkSe5ejj5u9w='} 2024-04-09T22:39:34.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6bfaddf696469bf-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:39:34.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:39:34.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:39:34.665 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:39:34.717 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:39:34.717 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:39:34.756 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:39:34.782 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702374.479594_726.1", 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:39:34.479594Z" 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:34.784 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:39:34.785 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:39:34.785 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:39:34.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:39:34.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:39:34.786 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:39:34.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:39:34.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:39:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:34.894+0000 7fa67c131780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:39:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:34.895+0000 7fa67c131780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:39:34.911 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:39:34.911 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:39:34.911 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:39:34.911 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:39:34.911 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:39:34.911 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:34.912 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:34.913 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702374.442860_719.1", 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:39:34.442860Z", 2024-04-09T22:39:34.914 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:39:34.915 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:39:34Z" 2024-04-09T22:39:34.916 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:39:34.917 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712702374.442860_719.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:39:34.442860Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:39:34Z" } 2024-04-09T22:39:34.917 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702374.442860_719.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:39:34.442860Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:39:34.917 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:39:34.917 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.442860_719.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:39:34.917 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712702374.479594_726.1 target=1_1712702374.442860_719.1 2024-04-09T22:39:39.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:39:39.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:39:39.919 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:39:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:39:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:39:40.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:40.029+0000 7fd14073e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:39:40.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:40.030+0000 7fd14073e780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:39:40.046 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:39:40.046 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:39:40.046 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:39:40.046 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:39:40.046 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:39:40.046 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:39:40.047 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:39:40.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702374.479594_726.1", 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:40.049 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:39:34.479594Z", 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:39:40.050 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:39:40.051 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:39:40Z" 2024-04-09T22:39:40.052 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:39:40.053 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712702374.479594_726.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:39:34.479594Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:39:40Z" } 2024-04-09T22:39:40.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702374.479594_726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:39:34.479594Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:39:40.053 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:39:40.053 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:39:40.054 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:39:40.054 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-50 2024-04-09T22:39:40.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:39:40.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:39:40.054 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:39:40.112 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:39:40.112 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:39:40.153 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-50 2024-04-09T22:39:40.171 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-50 2024-04-09T22:39:40.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:39:40.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:39:40.172 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:39:40.228 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:39:40.228 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:39:40.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:40.283+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:39:45.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:45.284+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:39:50.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:50.284+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:39:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:39:55.285+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:00.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:00.286+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:05.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:05.286+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:10.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:10.287+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:15.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:15.288+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:20.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:20.288+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:25.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:25.289+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:30.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:30.290+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:35.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:35.291+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:40.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:40.291+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:45.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:45.291+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:50.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:50.292+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:40:55.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:40:55.292+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:00.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:00.293+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:05.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:05.293+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:10.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:10.294+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:15.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:15.294+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:20.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:20.295+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:25.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:25.295+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:30.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:30.296+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:35.296+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:40.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:40.297+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:45.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:45.297+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:50.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:50.298+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:41:55.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:41:55.298+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:00.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:00.299+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:05.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:05.300+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:10.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:10.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:13.381 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:42:13.382+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:42:15.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:15.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:20.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:20.300+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:25.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:25.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:30.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:30.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:35.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:35.302+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:40.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:40.303+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:45.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:45.303+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:50.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:50.304+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:42:55.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:42:55.304+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:00.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:00.305+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:05.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:05.305+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:10.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:10.306+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:15.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:15.307+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:20.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:20.308+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:25.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:25.308+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:30.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:30.309+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:35.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:35.309+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:40.310+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:45.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:45.310+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:50.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:50.311+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:43:55.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:43:55.312+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:00.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:00.313+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:05.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:05.313+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:10.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:10.313+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:15.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:15.314+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:20.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:20.314+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:25.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:25.315+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:30.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:30.315+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-09T22:44:35.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:35.316+0000 7f19aef4d780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T22:44:35.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:35.316+0000 7f19aef4d780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:44:35.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:35.316+0000 7f19aef4d780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:44:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T22:44:35.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-04-09T22:44:35.344 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-52 2024-04-09T22:44:35.344 DEBUG:boto:path=/xryqqk-52/ 2024-04-09T22:44:35.344 DEBUG:boto:auth_path=/xryqqk-52/ 2024-04-09T22:44:35.345 DEBUG:boto:Method: PUT 2024-04-09T22:44:35.345 DEBUG:boto:Path: /xryqqk-52/ 2024-04-09T22:44:35.345 DEBUG:boto:Data: 2024-04-09T22:44:35.345 DEBUG:boto:Headers: {} 2024-04-09T22:44:35.345 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:35.345 DEBUG:boto:Port: 8000 2024-04-09T22:44:35.345 DEBUG:boto:Params: {} 2024-04-09T22:44:35.346 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:44:35.346 DEBUG:boto:Token: None 2024-04-09T22:44:35.346 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:44:35 GMT /xryqqk-52/ 2024-04-09T22:44:35.346 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jIT6FJ1sbYUczUOjGoZTnMO9RuE= 2024-04-09T22:44:35.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jIT6FJ1sbYUczUOjGoZTnMO9RuE=', 'Content-Length': '0'} 2024-04-09T22:44:35.373 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d054849f78618fd-006615c4d3-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:44:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:35.373 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-53 2024-04-09T22:44:35.373 DEBUG:boto:path=/xryqqk-53/ 2024-04-09T22:44:35.373 DEBUG:boto:auth_path=/xryqqk-53/ 2024-04-09T22:44:35.373 DEBUG:boto:Method: PUT 2024-04-09T22:44:35.373 DEBUG:boto:Path: /xryqqk-53/ 2024-04-09T22:44:35.374 DEBUG:boto:Data: 2024-04-09T22:44:35.374 DEBUG:boto:Headers: {} 2024-04-09T22:44:35.374 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:44:35.374 DEBUG:boto:Port: 8001 2024-04-09T22:44:35.374 DEBUG:boto:Params: {} 2024-04-09T22:44:35.377 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:44:35.377 DEBUG:boto:Token: None 2024-04-09T22:44:35.377 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:44:35 GMT /xryqqk-53/ 2024-04-09T22:44:35.377 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WJeUinx6DvYGq+3nZqYXjAPYwhI= 2024-04-09T22:44:35.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WJeUinx6DvYGq+3nZqYXjAPYwhI=', 'Content-Length': '0'} 2024-04-09T22:44:35.416 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6f52863568b601e-006615c4d3-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:44:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:35.417 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:35.417 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:35.417 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploads 2024-04-09T22:44:35.417 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploads 2024-04-09T22:44:35.417 DEBUG:boto:Method: POST 2024-04-09T22:44:35.417 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploads 2024-04-09T22:44:35.417 DEBUG:boto:Data: 2024-04-09T22:44:35.418 DEBUG:boto:Headers: {} 2024-04-09T22:44:35.418 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:35.418 DEBUG:boto:Port: 8000 2024-04-09T22:44:35.418 DEBUG:boto:Params: {} 2024-04-09T22:44:35.418 DEBUG:boto:Token: None 2024-04-09T22:44:35.418 DEBUG:boto:StringToSign: POST Tue, 09 Apr 2024 22:44:35 GMT /xryqqk-52/MULTIPART?uploads 2024-04-09T22:44:35.419 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fxulVOH4uDLtWqBpeag1DuxrBgI= 2024-04-09T22:44:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fxulVOH4uDLtWqBpeag1DuxrBgI=', 'Content-Length': '0'} 2024-04-09T22:44:37.045 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009893d3443fd98759-006615c4d3-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.045 DEBUG:boto:b'xryqqk-52MULTIPART2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW' 2024-04-09T22:44:37.076 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.077 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.077 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=1 2024-04-09T22:44:37.077 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=1 2024-04-09T22:44:37.077 DEBUG:boto:Method: PUT 2024-04-09T22:44:37.077 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=1 2024-04-09T22:44:37.077 DEBUG:boto:Data: 2024-04-09T22:44:37.077 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-09T22:44:37.077 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:37.077 DEBUG:boto:Port: 8000 2024-04-09T22:44:37.077 DEBUG:boto:Params: {} 2024-04-09T22:44:37.078 DEBUG:boto:Token: None 2024-04-09T22:44:37.078 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Tue, 09 Apr 2024 22:44:37 GMT /xryqqk-52/MULTIPART?partNumber=1&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.078 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cq6IDxefIVFXta/yfS49/BGflD4= 2024-04-09T22:44:37.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cq6IDxefIVFXta/yfS49/BGflD4='} 2024-04-09T22:44:37.116 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc63dc1f8e85d646-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.147 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.147 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.147 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=2 2024-04-09T22:44:37.147 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=2 2024-04-09T22:44:37.147 DEBUG:boto:Method: PUT 2024-04-09T22:44:37.147 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=2 2024-04-09T22:44:37.147 DEBUG:boto:Data: 2024-04-09T22:44:37.148 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-09T22:44:37.148 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:37.148 DEBUG:boto:Port: 8000 2024-04-09T22:44:37.148 DEBUG:boto:Params: {} 2024-04-09T22:44:37.148 DEBUG:boto:Token: None 2024-04-09T22:44:37.148 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Tue, 09 Apr 2024 22:44:37 GMT /xryqqk-52/MULTIPART?partNumber=2&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.148 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7w8CLDcI67b2TbeHFiOZ1UPA/Z0= 2024-04-09T22:44:37.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7w8CLDcI67b2TbeHFiOZ1UPA/Z0='} 2024-04-09T22:44:37.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fe10f8041fd8e95-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.200 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.200 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.200 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=3 2024-04-09T22:44:37.200 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=3 2024-04-09T22:44:37.200 DEBUG:boto:Method: PUT 2024-04-09T22:44:37.200 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=3 2024-04-09T22:44:37.201 DEBUG:boto:Data: 2024-04-09T22:44:37.201 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-09T22:44:37.201 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:37.201 DEBUG:boto:Port: 8000 2024-04-09T22:44:37.201 DEBUG:boto:Params: {} 2024-04-09T22:44:37.201 DEBUG:boto:Token: None 2024-04-09T22:44:37.201 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Tue, 09 Apr 2024 22:44:37 GMT /xryqqk-52/MULTIPART?partNumber=3&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.201 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:J3cVPJ/AP1WpT1wberONa1SWw0Q= 2024-04-09T22:44:37.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J3cVPJ/AP1WpT1wberONa1SWw0Q='} 2024-04-09T22:44:37.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3ded15ac38d7f1e-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.252 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.253 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.253 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=4 2024-04-09T22:44:37.253 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=4 2024-04-09T22:44:37.253 DEBUG:boto:Method: PUT 2024-04-09T22:44:37.253 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=4 2024-04-09T22:44:37.253 DEBUG:boto:Data: 2024-04-09T22:44:37.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-09T22:44:37.253 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:37.253 DEBUG:boto:Port: 8000 2024-04-09T22:44:37.254 DEBUG:boto:Params: {} 2024-04-09T22:44:37.254 DEBUG:boto:Token: None 2024-04-09T22:44:37.254 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Tue, 09 Apr 2024 22:44:37 GMT /xryqqk-52/MULTIPART?partNumber=4&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.254 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EY3ZhnzyF8+CkAOoBP4mW54eZx0= 2024-04-09T22:44:37.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EY3ZhnzyF8+CkAOoBP4mW54eZx0='} 2024-04-09T22:44:37.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a80e22f7cfffca7-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.286 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.286 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.286 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.286 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.286 DEBUG:boto:Method: GET 2024-04-09T22:44:37.286 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.286 DEBUG:boto:Data: 2024-04-09T22:44:37.287 DEBUG:boto:Headers: {} 2024-04-09T22:44:37.287 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:37.287 DEBUG:boto:Port: 8000 2024-04-09T22:44:37.287 DEBUG:boto:Params: {} 2024-04-09T22:44:37.287 DEBUG:boto:Token: None 2024-04-09T22:44:37.287 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 22:44:37 GMT /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.287 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2jDxbPYwds8+T3ijCTLRTnzylj4= 2024-04-09T22:44:37.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2jDxbPYwds8+T3ijCTLRTnzylj4=', 'Content-Length': '0'} 2024-04-09T22:44:37.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000984b0b3a6c72962a-006615c4d5-1b6d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.331 DEBUG:boto:path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.331 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART 2024-04-09T22:44:37.331 DEBUG:boto:path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.331 DEBUG:boto:auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.331 DEBUG:boto:Method: POST 2024-04-09T22:44:37.331 DEBUG:boto:Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.331 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-04-09T22:44:37.331 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-04-09T22:44:37.331 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:44:37.331 DEBUG:boto:Port: 8000 2024-04-09T22:44:37.332 DEBUG:boto:Params: {} 2024-04-09T22:44:37.332 DEBUG:boto:Token: None 2024-04-09T22:44:37.332 DEBUG:boto:StringToSign: POST text/xml Tue, 09 Apr 2024 22:44:37 GMT /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-09T22:44:37.332 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uLluuPnzl2KeAiIPGZJxSknqK2U= 2024-04-09T22:44:37.332 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uLluuPnzl2KeAiIPGZJxSknqK2U=', 'Content-Length': '460'} 2024-04-09T22:44:37.342 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5fd54acbd92cf77-006615c4d5-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:44:37.342 DEBUG:boto:http://smithi063.front.sepia.ceph.com:8000/xryqqk-52/MULTIPARTxryqqk-52MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-09T22:44:37.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:44:37.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:44:37.343 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:44:37.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:44:37.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:44:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702675.402940_775.1", 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:44:35.402940Z" 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:44:37.468 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702675.402940_775.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:44:37.468 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:44:37.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:44:37.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:44:37.469 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:44:37.518 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:44:37.518 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:44:37.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:37.571+0000 7f5ecad85780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:44:37.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:37.572+0000 7f5ecad85780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:44:37.589 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:44:37.589 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:44:37.589 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:44:37.589 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:44:37.590 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:44:37.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702675.402940_775.1", 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:44:37.593 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:44:35.402940Z", 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:44:37.594 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:44:37.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.596 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:44:37.596 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:44:37.596 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:44:37.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:44:37.596 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:44:37Z" 2024-04-09T22:44:37.596 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:44:37.597 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712702675.402940_775.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:44:35.402940Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:44:37Z" } 2024-04-09T22:44:37.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702675.402940_775.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:44:35.402940Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:44:37.597 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702675.402940_775.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:44:37.598 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702675.402940_775.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:44:37.598 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:44:37.598 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-52 2024-04-09T22:44:37.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:44:37.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:44:37.598 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:44:37.695 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-52 2024-04-09T22:44:37.717 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-52 2024-04-09T22:44:37.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:44:37.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:44:37.718 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-52 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:44:37.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:44:37.774 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:44:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:37.836+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:44:42.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:42.837+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:44:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:47.837+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:44:52.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:52.838+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:44:57.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:44:57.839+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:02.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:02.839+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:07.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:07.840+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:12.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:12.840+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:17.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:17.840+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:22.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:22.841+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:27.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:27.842+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:32.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:32.843+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:37.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:37.844+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:42.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:42.843+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:47.844+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:52.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:52.845+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:45:57.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:45:57.845+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:02.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:02.846+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:07.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:07.846+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:12.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:12.847+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:17.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:17.847+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:22.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:22.848+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:27.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:27.848+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:32.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:32.849+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:37.849+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:42.850+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:47.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:47.851+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:52.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:52.851+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:46:57.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:46:57.852+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:02.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:02.853+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:07.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:07.853+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:12.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:12.854+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:17.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:17.855+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:22.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:22.854+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:27.855+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:32.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:32.855+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:33.387 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:47:33.387+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:47:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:37.856+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:42.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:42.856+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:47.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:47.857+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:52.857+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:47:57.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:47:57.858+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:02.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:02.858+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:07.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:07.859+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:12.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:12.860+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:17.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:17.860+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:22.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:22.861+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:27.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:27.861+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:32.862+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:37.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:37.863+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:42.863+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:47.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:47.864+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:52.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:52.865+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:48:57.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:48:57.866+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:02.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:02.865+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:07.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:07.866+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:12.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:12.866+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:17.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:17.867+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:22.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:22.867+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:27.868+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-09T22:49:32.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:32.868+0000 7f5e4d9d1780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T22:49:32.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:32.868+0000 7f5e4d9d1780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:49:32.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:32.868+0000 7f5e4d9d1780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:49:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T22:49:32.894 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-04-09T22:49:32.895 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-54 2024-04-09T22:49:32.896 DEBUG:boto:path=/xryqqk-54/ 2024-04-09T22:49:32.896 DEBUG:boto:auth_path=/xryqqk-54/ 2024-04-09T22:49:32.896 DEBUG:boto:Method: PUT 2024-04-09T22:49:32.896 DEBUG:boto:Path: /xryqqk-54/ 2024-04-09T22:49:32.896 DEBUG:boto:Data: 2024-04-09T22:49:32.896 DEBUG:boto:Headers: {} 2024-04-09T22:49:32.896 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:32.896 DEBUG:boto:Port: 8000 2024-04-09T22:49:32.897 DEBUG:boto:Params: {} 2024-04-09T22:49:32.897 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:49:32.897 DEBUG:boto:Token: None 2024-04-09T22:49:32.897 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:49:32 GMT /xryqqk-54/ 2024-04-09T22:49:32.898 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k+bqStpueuY03qZzzvaMh2XJPPY= 2024-04-09T22:49:32.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k+bqStpueuY03qZzzvaMh2XJPPY=', 'Content-Length': '0'} 2024-04-09T22:49:32.933 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000659d350fd199f573-006615c5fc-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:32.934 DEBUG:boto:path=/xryqqk-54/testobj-sse-c 2024-04-09T22:49:32.934 DEBUG:boto:auth_path=/xryqqk-54/testobj-sse-c 2024-04-09T22:49:32.934 DEBUG:boto:Method: PUT 2024-04-09T22:49:32.934 DEBUG:boto:Path: /xryqqk-54/testobj-sse-c 2024-04-09T22:49:32.934 DEBUG:boto:Data: 2024-04-09T22:49:32.934 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-09T22:49:32.935 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:32.935 DEBUG:boto:Port: 8000 2024-04-09T22:49:32.935 DEBUG:boto:Params: {} 2024-04-09T22:49:32.935 DEBUG:boto:Token: None 2024-04-09T22:49:32.935 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 09 Apr 2024 22:49:32 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /xryqqk-54/testobj-sse-c 2024-04-09T22:49:32.935 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OC1L02YjI3oQSuLzEh5+wDilYT0= 2024-04-09T22:49:32.935 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OC1L02YjI3oQSuLzEh5+wDilYT0='} 2024-04-09T22:49:32.948 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000004d5d0b6cf9bfedc-006615c5fc-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:32.948 DEBUG:boto:path=/xryqqk-54/testobj-sse-kms 2024-04-09T22:49:32.948 DEBUG:boto:auth_path=/xryqqk-54/testobj-sse-kms 2024-04-09T22:49:32.948 DEBUG:boto:Method: PUT 2024-04-09T22:49:32.948 DEBUG:boto:Path: /xryqqk-54/testobj-sse-kms 2024-04-09T22:49:32.949 DEBUG:boto:Data: 2024-04-09T22:49:32.949 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-09T22:49:32.949 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:32.949 DEBUG:boto:Port: 8000 2024-04-09T22:49:32.949 DEBUG:boto:Params: {} 2024-04-09T22:49:32.949 DEBUG:boto:Token: None 2024-04-09T22:49:32.949 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Tue, 09 Apr 2024 22:49:32 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /xryqqk-54/testobj-sse-kms 2024-04-09T22:49:32.949 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UZSJ65V85toqhO6C96SdsnDjhQc= 2024-04-09T22:49:32.949 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UZSJ65V85toqhO6C96SdsnDjhQc='} 2024-04-09T22:49:32.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6eHJ5cXFrLTU0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000000d154ce0ae7f6976-006615c5fc-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:32.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:32.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:32.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:49:33.014 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:33.014 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:33.061 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:49:33.078 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702972.930686_822.1", 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:49:32.930686Z" 2024-04-09T22:49:33.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:49:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:49:33.080 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.080 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:49:33.081 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702972.930686_822.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:33.081 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:49:33.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:33.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:33.082 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:49:33.130 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:33.131 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:33.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:33.165+0000 7f453bf35780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:49:33.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:33.166+0000 7f453bf35780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:49:33.183 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:49:33.183 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:49:33.183 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:49:33.183 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:49:33.183 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:49:33.183 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:49:33.184 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:33.185 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702972.930686_822.1", 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.186 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:49:32.930686Z", 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.187 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.188 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:49:33Z" 2024-04-09T22:49:33.189 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:49:33.189 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712702972.930686_822.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:49:32.930686Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:49:33Z" } 2024-04-09T22:49:33.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702972.930686_822.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:49:32.930686Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:49:33.189 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702972.930686_822.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:33.189 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702972.930686_822.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:33.189 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:49:33.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:49:33.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:49:33.190 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:49:33.246 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:33.247 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:33.288 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-54 2024-04-09T22:49:33.311 DEBUG:boto:path=/xryqqk-54/ 2024-04-09T22:49:33.312 DEBUG:boto:auth_path=/xryqqk-54/ 2024-04-09T22:49:33.312 DEBUG:boto:Method: HEAD 2024-04-09T22:49:33.312 DEBUG:boto:Path: /xryqqk-54/ 2024-04-09T22:49:33.312 DEBUG:boto:Data: 2024-04-09T22:49:33.312 DEBUG:boto:Headers: {} 2024-04-09T22:49:33.312 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:49:33.312 DEBUG:boto:Port: 8001 2024-04-09T22:49:33.312 DEBUG:boto:Params: {} 2024-04-09T22:49:33.313 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T22:49:33.313 DEBUG:boto:Token: None 2024-04-09T22:49:33.313 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-54/ 2024-04-09T22:49:33.313 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3y7iItu0wmxzkSlgjr0Mq8ZNKbc= 2024-04-09T22:49:33.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3y7iItu0wmxzkSlgjr0Mq8ZNKbc=', 'Content-Length': '0'} 2024-04-09T22:49:33.317 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abcd83ed53ed6891-006615c5fd-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.317 DEBUG:boto:path=/xryqqk-54/testobj-sse-c 2024-04-09T22:49:33.317 DEBUG:boto:auth_path=/xryqqk-54/testobj-sse-c 2024-04-09T22:49:33.317 DEBUG:boto:Method: HEAD 2024-04-09T22:49:33.318 DEBUG:boto:Path: /xryqqk-54/testobj-sse-c 2024-04-09T22:49:33.318 DEBUG:boto:Data: 2024-04-09T22:49:33.318 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-09T22:49:33.318 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T22:49:33.318 DEBUG:boto:Port: 8001 2024-04-09T22:49:33.318 DEBUG:boto:Params: {} 2024-04-09T22:49:33.318 DEBUG:boto:Token: None 2024-04-09T22:49:33.318 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:49:33 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /xryqqk-54/testobj-sse-c 2024-04-09T22:49:33.318 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+I/xrDNwSO5R1gmQfwsiScJ8FCk= 2024-04-09T22:49:33.319 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+I/xrDNwSO5R1gmQfwsiScJ8FCk=', 'Content-Length': '0'} 2024-04-09T22:49:33.320 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000949d09b5515f12f2-006615c5fd-1b6f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2024-04-09T22:49:33.322 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-55 2024-04-09T22:49:33.322 DEBUG:boto:path=/xryqqk-55/ 2024-04-09T22:49:33.322 DEBUG:boto:auth_path=/xryqqk-55/ 2024-04-09T22:49:33.322 DEBUG:boto:Method: PUT 2024-04-09T22:49:33.322 DEBUG:boto:Path: /xryqqk-55/ 2024-04-09T22:49:33.322 DEBUG:boto:Data: 2024-04-09T22:49:33.322 DEBUG:boto:Headers: {} 2024-04-09T22:49:33.322 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.322 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.322 DEBUG:boto:Params: {} 2024-04-09T22:49:33.323 DEBUG:boto:Token: None 2024-04-09T22:49:33.323 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/ 2024-04-09T22:49:33.323 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7AablSRZxA0ofierpiO6O4CcbBM= 2024-04-09T22:49:33.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7AablSRZxA0ofierpiO6O4CcbBM=', 'Content-Length': '0'} 2024-04-09T22:49:33.353 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000152cb456ff34fd84-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.353 DEBUG:boto:path=/xryqqk-55/ 2024-04-09T22:49:33.353 DEBUG:boto:auth_path=/xryqqk-55/ 2024-04-09T22:49:33.353 DEBUG:boto:Method: HEAD 2024-04-09T22:49:33.353 DEBUG:boto:Path: /xryqqk-55/ 2024-04-09T22:49:33.353 DEBUG:boto:Data: 2024-04-09T22:49:33.353 DEBUG:boto:Headers: {} 2024-04-09T22:49:33.354 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.354 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.354 DEBUG:boto:Params: {} 2024-04-09T22:49:33.354 DEBUG:boto:Token: None 2024-04-09T22:49:33.354 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/ 2024-04-09T22:49:33.354 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-09T22:49:33.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-09T22:49:33.356 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8bf2e4adbf6df6f-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.357 DEBUG:boto:path=/xryqqk-55/a 2024-04-09T22:49:33.357 DEBUG:boto:auth_path=/xryqqk-55/a 2024-04-09T22:49:33.357 DEBUG:boto:Method: PUT 2024-04-09T22:49:33.357 DEBUG:boto:Path: /xryqqk-55/a 2024-04-09T22:49:33.357 DEBUG:boto:Data: 2024-04-09T22:49:33.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:49:33.357 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.357 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.358 DEBUG:boto:Params: {} 2024-04-09T22:49:33.358 DEBUG:boto:Token: None 2024-04-09T22:49:33.358 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/a 2024-04-09T22:49:33.358 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dsLHuDD2EEfCSKN3tiv1cZtLxXE= 2024-04-09T22:49:33.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dsLHuDD2EEfCSKN3tiv1cZtLxXE='} 2024-04-09T22:49:33.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bccf8cc5490a9073-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.403 DEBUG:boto:path=/xryqqk-55/ 2024-04-09T22:49:33.403 DEBUG:boto:auth_path=/xryqqk-55/ 2024-04-09T22:49:33.403 DEBUG:boto:Method: HEAD 2024-04-09T22:49:33.403 DEBUG:boto:Path: /xryqqk-55/ 2024-04-09T22:49:33.403 DEBUG:boto:Data: 2024-04-09T22:49:33.403 DEBUG:boto:Headers: {} 2024-04-09T22:49:33.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.404 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.404 DEBUG:boto:Params: {} 2024-04-09T22:49:33.404 DEBUG:boto:Token: None 2024-04-09T22:49:33.404 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/ 2024-04-09T22:49:33.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-09T22:49:33.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-09T22:49:33.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000108d6f58a8dccbb5-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.406 DEBUG:boto:path=/xryqqk-55/b 2024-04-09T22:49:33.407 DEBUG:boto:auth_path=/xryqqk-55/b 2024-04-09T22:49:33.407 DEBUG:boto:Method: PUT 2024-04-09T22:49:33.407 DEBUG:boto:Path: /xryqqk-55/b 2024-04-09T22:49:33.407 DEBUG:boto:Data: 2024-04-09T22:49:33.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:49:33.407 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.407 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.407 DEBUG:boto:Params: {} 2024-04-09T22:49:33.407 DEBUG:boto:Token: None 2024-04-09T22:49:33.408 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/b 2024-04-09T22:49:33.408 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QaB7ldOJnu2/yWFh268AA/8hL8Y= 2024-04-09T22:49:33.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QaB7ldOJnu2/yWFh268AA/8hL8Y='} 2024-04-09T22:49:33.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a6597b6be01f3a1-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.416 DEBUG:boto:path=/xryqqk-55/ 2024-04-09T22:49:33.416 DEBUG:boto:auth_path=/xryqqk-55/ 2024-04-09T22:49:33.416 DEBUG:boto:Method: HEAD 2024-04-09T22:49:33.416 DEBUG:boto:Path: /xryqqk-55/ 2024-04-09T22:49:33.416 DEBUG:boto:Data: 2024-04-09T22:49:33.416 DEBUG:boto:Headers: {} 2024-04-09T22:49:33.416 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.416 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.416 DEBUG:boto:Params: {} 2024-04-09T22:49:33.417 DEBUG:boto:Token: None 2024-04-09T22:49:33.417 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/ 2024-04-09T22:49:33.417 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-09T22:49:33.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-09T22:49:33.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2d9d387ea392614-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.419 DEBUG:boto:path=/xryqqk-55/c 2024-04-09T22:49:33.419 DEBUG:boto:auth_path=/xryqqk-55/c 2024-04-09T22:49:33.419 DEBUG:boto:Method: PUT 2024-04-09T22:49:33.420 DEBUG:boto:Path: /xryqqk-55/c 2024-04-09T22:49:33.420 DEBUG:boto:Data: 2024-04-09T22:49:33.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:49:33.420 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.420 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.420 DEBUG:boto:Params: {} 2024-04-09T22:49:33.420 DEBUG:boto:Token: None 2024-04-09T22:49:33.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/c 2024-04-09T22:49:33.420 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jluzeo3BD4LQtpkTkt12cs/xr4U= 2024-04-09T22:49:33.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jluzeo3BD4LQtpkTkt12cs/xr4U='} 2024-04-09T22:49:33.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c05f02a439d9a97-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.463 DEBUG:boto:path=/xryqqk-55/ 2024-04-09T22:49:33.463 DEBUG:boto:auth_path=/xryqqk-55/ 2024-04-09T22:49:33.463 DEBUG:boto:Method: HEAD 2024-04-09T22:49:33.463 DEBUG:boto:Path: /xryqqk-55/ 2024-04-09T22:49:33.463 DEBUG:boto:Data: 2024-04-09T22:49:33.463 DEBUG:boto:Headers: {} 2024-04-09T22:49:33.463 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.463 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.464 DEBUG:boto:Params: {} 2024-04-09T22:49:33.464 DEBUG:boto:Token: None 2024-04-09T22:49:33.464 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/ 2024-04-09T22:49:33.464 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-09T22:49:33.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-09T22:49:33.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081b23efab3c2b47c-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.467 DEBUG:boto:path=/xryqqk-55/d 2024-04-09T22:49:33.467 DEBUG:boto:auth_path=/xryqqk-55/d 2024-04-09T22:49:33.467 DEBUG:boto:Method: PUT 2024-04-09T22:49:33.467 DEBUG:boto:Path: /xryqqk-55/d 2024-04-09T22:49:33.467 DEBUG:boto:Data: 2024-04-09T22:49:33.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:49:33.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:49:33.467 DEBUG:boto:Port: 8000 2024-04-09T22:49:33.467 DEBUG:boto:Params: {} 2024-04-09T22:49:33.468 DEBUG:boto:Token: None 2024-04-09T22:49:33.468 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:49:33 GMT /xryqqk-55/d 2024-04-09T22:49:33.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:31dbqPlxGsL+JI3cOAgB9dxiJIQ= 2024-04-09T22:49:33.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:31dbqPlxGsL+JI3cOAgB9dxiJIQ='} 2024-04-09T22:49:33.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa0f3f5295d357fa-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:49:33.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:33.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:33.475 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:49:33.534 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:33.534 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:33.572 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:49:33.599 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702973.352006_826.1", 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:49:33.352006Z" 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.600 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:49:33.601 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:33.602 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:49:33.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:33.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:33.602 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:49:33.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:33.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:33.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:33.690+0000 7fe5e6021780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:49:33.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:33.691+0000 7fe5e6021780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.712 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:49:33.713 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:49:33.714 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.715 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702973.335445_823.1", 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:49:33.335445Z", 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:33.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:49:33Z" 2024-04-09T22:49:33.718 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:49:33.719 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712702973.335445_823.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:49:33.335445Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:49:33Z" } 2024-04-09T22:49:33.719 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702973.335445_823.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:49:33.335445Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:49:33.720 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:33.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.335445_823.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:33.720 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712702973.352006_826.1 target=1_1712702973.335445_823.1 2024-04-09T22:49:38.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:38.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:49:38.721 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:49:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:38.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:38.822+0000 7fc1734a0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:49:38.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:38.823+0000 7fc1734a0780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:49:38.838 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:49:38.838 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:49:38.839 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:38.840 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:38.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712702973.352006_826.1", 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:49:33.352006Z", 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:38.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:49:38.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:49:38Z" 2024-04-09T22:49:38.844 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:49:38.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712702973.352006_826.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:49:33.352006Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:49:38Z" } 2024-04-09T22:49:38.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702973.352006_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:49:33.352006Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:49:38.846 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:38.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:49:38.846 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:49:38.846 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-55 2024-04-09T22:49:38.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:49:38.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:49:38.846 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:49:38.904 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:38.904 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:38.952 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-55 2024-04-09T22:49:38.974 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-55 2024-04-09T22:49:38.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:49:38.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:49:38.975 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:49:39.032 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:49:39.032 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:49:39.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:39.076+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:49:44.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:44.077+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:49:49.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:49.077+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:49:54.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:54.078+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:49:59.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:49:59.079+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:04.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:04.079+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:09.080+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:14.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:14.080+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:19.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:19.081+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:24.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:24.082+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:29.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:29.082+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:34.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:34.083+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:39.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:39.084+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:44.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:44.084+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:49.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:49.084+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:54.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:54.085+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:50:59.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:50:59.086+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:04.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:04.086+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:09.086+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:14.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:14.087+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:19.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:19.088+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:24.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:24.088+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:29.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:29.089+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:34.089+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:39.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:39.090+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:44.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:44.090+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:49.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:49.091+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:54.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:54.091+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:51:59.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:51:59.092+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:04.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:04.092+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:09.093+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:14.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:14.094+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:19.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:19.095+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:24.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:24.094+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:29.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:29.095+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:33.390 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:52:33.390+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:52:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:34.095+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:39.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:39.096+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:44.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:44.096+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:49.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:49.096+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:54.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:54.097+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:52:59.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:52:59.098+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:04.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:04.099+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:09.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:09.099+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:14.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:14.100+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:19.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:19.101+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:24.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:24.102+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:29.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:29.102+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:34.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:34.103+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:39.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:39.103+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:44.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:44.104+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:49.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:49.105+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:54.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:53:59.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:53:59.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:04.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:04.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:09.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:09.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:14.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:14.107+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:19.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:19.108+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:24.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:24.108+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:29.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:29.109+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-09T22:54:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:34.110+0000 7ff6276da780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T22:54:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:34.110+0000 7ff6276da780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:54:34.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:34.110+0000 7ff6276da780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:54:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T22:54:34.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2024-04-09T22:54:34.154 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-56 2024-04-09T22:54:34.154 DEBUG:boto:path=/xryqqk-56/ 2024-04-09T22:54:34.154 DEBUG:boto:auth_path=/xryqqk-56/ 2024-04-09T22:54:34.155 DEBUG:boto:Method: PUT 2024-04-09T22:54:34.155 DEBUG:boto:Path: /xryqqk-56/ 2024-04-09T22:54:34.155 DEBUG:boto:Data: 2024-04-09T22:54:34.155 DEBUG:boto:Headers: {} 2024-04-09T22:54:34.155 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.155 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.155 DEBUG:boto:Params: {} 2024-04-09T22:54:34.155 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:54:34.156 DEBUG:boto:Token: None 2024-04-09T22:54:34.156 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/ 2024-04-09T22:54:34.156 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uVaePOb/pA0dF1rNaoJQ9MlENG0= 2024-04-09T22:54:34.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uVaePOb/pA0dF1rNaoJQ9MlENG0=', 'Content-Length': '0'} 2024-04-09T22:54:34.179 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000049d5522180b23483-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.179 DEBUG:boto:path=/xryqqk-56/ 2024-04-09T22:54:34.180 DEBUG:boto:auth_path=/xryqqk-56/ 2024-04-09T22:54:34.180 DEBUG:boto:Method: HEAD 2024-04-09T22:54:34.180 DEBUG:boto:Path: /xryqqk-56/ 2024-04-09T22:54:34.180 DEBUG:boto:Data: 2024-04-09T22:54:34.180 DEBUG:boto:Headers: {} 2024-04-09T22:54:34.180 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.180 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.180 DEBUG:boto:Params: {} 2024-04-09T22:54:34.180 DEBUG:boto:Token: None 2024-04-09T22:54:34.180 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/ 2024-04-09T22:54:34.181 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-09T22:54:34.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-09T22:54:34.183 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efdd6fbe26344f18-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.183 DEBUG:boto:path=/xryqqk-56/a 2024-04-09T22:54:34.183 DEBUG:boto:auth_path=/xryqqk-56/a 2024-04-09T22:54:34.183 DEBUG:boto:Method: PUT 2024-04-09T22:54:34.183 DEBUG:boto:Path: /xryqqk-56/a 2024-04-09T22:54:34.183 DEBUG:boto:Data: 2024-04-09T22:54:34.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:54:34.184 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.184 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.184 DEBUG:boto:Params: {} 2024-04-09T22:54:34.184 DEBUG:boto:Token: None 2024-04-09T22:54:34.184 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/a 2024-04-09T22:54:34.184 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YXHIrAS4GLSjb6FGhFDXbq/BNPI= 2024-04-09T22:54:34.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YXHIrAS4GLSjb6FGhFDXbq/BNPI='} 2024-04-09T22:54:34.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbf7aa6511cbffd6-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.191 DEBUG:boto:path=/xryqqk-56/ 2024-04-09T22:54:34.191 DEBUG:boto:auth_path=/xryqqk-56/ 2024-04-09T22:54:34.191 DEBUG:boto:Method: HEAD 2024-04-09T22:54:34.191 DEBUG:boto:Path: /xryqqk-56/ 2024-04-09T22:54:34.191 DEBUG:boto:Data: 2024-04-09T22:54:34.192 DEBUG:boto:Headers: {} 2024-04-09T22:54:34.192 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.192 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.192 DEBUG:boto:Params: {} 2024-04-09T22:54:34.192 DEBUG:boto:Token: None 2024-04-09T22:54:34.192 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/ 2024-04-09T22:54:34.192 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-09T22:54:34.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-09T22:54:34.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be35ba9afa081219-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.195 DEBUG:boto:path=/xryqqk-56/b 2024-04-09T22:54:34.195 DEBUG:boto:auth_path=/xryqqk-56/b 2024-04-09T22:54:34.195 DEBUG:boto:Method: PUT 2024-04-09T22:54:34.195 DEBUG:boto:Path: /xryqqk-56/b 2024-04-09T22:54:34.195 DEBUG:boto:Data: 2024-04-09T22:54:34.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:54:34.195 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.195 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.195 DEBUG:boto:Params: {} 2024-04-09T22:54:34.195 DEBUG:boto:Token: None 2024-04-09T22:54:34.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/b 2024-04-09T22:54:34.196 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XFXuCeJm1ROoJHqR8M7VY7il4NM= 2024-04-09T22:54:34.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XFXuCeJm1ROoJHqR8M7VY7il4NM='} 2024-04-09T22:54:34.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000537b62a3452589fd-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.239 DEBUG:boto:path=/xryqqk-56/ 2024-04-09T22:54:34.239 DEBUG:boto:auth_path=/xryqqk-56/ 2024-04-09T22:54:34.239 DEBUG:boto:Method: HEAD 2024-04-09T22:54:34.239 DEBUG:boto:Path: /xryqqk-56/ 2024-04-09T22:54:34.239 DEBUG:boto:Data: 2024-04-09T22:54:34.239 DEBUG:boto:Headers: {} 2024-04-09T22:54:34.239 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.239 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.239 DEBUG:boto:Params: {} 2024-04-09T22:54:34.240 DEBUG:boto:Token: None 2024-04-09T22:54:34.240 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/ 2024-04-09T22:54:34.240 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-09T22:54:34.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-09T22:54:34.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002152c1b57b0c8471-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.242 DEBUG:boto:path=/xryqqk-56/c 2024-04-09T22:54:34.242 DEBUG:boto:auth_path=/xryqqk-56/c 2024-04-09T22:54:34.243 DEBUG:boto:Method: PUT 2024-04-09T22:54:34.243 DEBUG:boto:Path: /xryqqk-56/c 2024-04-09T22:54:34.243 DEBUG:boto:Data: 2024-04-09T22:54:34.243 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:54:34.243 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.243 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.243 DEBUG:boto:Params: {} 2024-04-09T22:54:34.243 DEBUG:boto:Token: None 2024-04-09T22:54:34.243 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/c 2024-04-09T22:54:34.244 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zySh4Lh4wlYBKDuXrjl8WexxW5s= 2024-04-09T22:54:34.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zySh4Lh4wlYBKDuXrjl8WexxW5s='} 2024-04-09T22:54:34.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbd73fa971cef15c-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.250 DEBUG:boto:path=/xryqqk-56/ 2024-04-09T22:54:34.250 DEBUG:boto:auth_path=/xryqqk-56/ 2024-04-09T22:54:34.250 DEBUG:boto:Method: HEAD 2024-04-09T22:54:34.250 DEBUG:boto:Path: /xryqqk-56/ 2024-04-09T22:54:34.250 DEBUG:boto:Data: 2024-04-09T22:54:34.251 DEBUG:boto:Headers: {} 2024-04-09T22:54:34.251 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.251 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.251 DEBUG:boto:Params: {} 2024-04-09T22:54:34.251 DEBUG:boto:Token: None 2024-04-09T22:54:34.251 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/ 2024-04-09T22:54:34.251 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-09T22:54:34.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-09T22:54:34.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac73ead0aaad4fd9-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.253 DEBUG:boto:path=/xryqqk-56/d 2024-04-09T22:54:34.254 DEBUG:boto:auth_path=/xryqqk-56/d 2024-04-09T22:54:34.254 DEBUG:boto:Method: PUT 2024-04-09T22:54:34.254 DEBUG:boto:Path: /xryqqk-56/d 2024-04-09T22:54:34.254 DEBUG:boto:Data: 2024-04-09T22:54:34.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:54:34.254 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:54:34.254 DEBUG:boto:Port: 8000 2024-04-09T22:54:34.254 DEBUG:boto:Params: {} 2024-04-09T22:54:34.254 DEBUG:boto:Token: None 2024-04-09T22:54:34.254 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:54:34 GMT /xryqqk-56/d 2024-04-09T22:54:34.255 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tGTuH680e5BMsuq0tQpLbFRbnD8= 2024-04-09T22:54:34.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tGTuH680e5BMsuq0tQpLbFRbnD8='} 2024-04-09T22:54:34.260 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f55b9d0b4ddc15a2-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:54:34.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:54:34.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:54:34.261 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:54:34.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:54:34.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:54:34.641 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:54:34.656 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703274.178131_867.1", 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:54:34.178131Z" 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:54:34.657 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:54:34.658 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703274.178131_867.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:54:34.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:54:34.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:54:34.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:54:34.659 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:54:34.710 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:54:34.710 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:54:34.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:34.749+0000 7fe28dc16780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:54:34.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:34.750+0000 7fe28dc16780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:54:34.773 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:54:34.773 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:54:34.773 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:54:34.773 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:54:34.773 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:54:34.773 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:54:34.774 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:54:34.775 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703274.178131_867.1", 2024-04-09T22:54:34.776 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:54:34.178131Z", 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:54:34.777 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:54:34.778 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:54:34Z" 2024-04-09T22:54:34.779 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:54:34.780 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712703274.178131_867.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:54:34.178131Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:54:34Z" } 2024-04-09T22:54:34.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703274.178131_867.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:54:34.178131Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:54:34.781 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703274.178131_867.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:54:34.781 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703274.178131_867.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:54:34.781 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:54:34.781 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-56 2024-04-09T22:54:34.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:54:34.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:54:34.782 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:54:34.897 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:54:34.897 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:54:34.949 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-56 2024-04-09T22:54:34.971 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-56 2024-04-09T22:54:34.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:54:34.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:54:34.972 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:54:35.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:54:35.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:54:35.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:35.085+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:54:40.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:40.086+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:54:45.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:45.086+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:54:50.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:50.087+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:54:55.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:54:55.087+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:00.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:00.088+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:05.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:05.089+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:10.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:10.089+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:15.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:15.090+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:20.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:20.090+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:25.091+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:30.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:30.092+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:35.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:35.093+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:40.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:40.094+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:45.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:45.093+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:50.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:50.094+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:55:55.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:55:55.094+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:00.095+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:05.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:05.095+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:10.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:10.096+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:15.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:15.096+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:20.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:20.097+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:25.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:25.097+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:30.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:30.098+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:35.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:35.099+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:40.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:40.099+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:45.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:45.100+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:50.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:50.100+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:56:55.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:56:55.101+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:00.102+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:05.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:05.102+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:10.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:10.103+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:15.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:15.104+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:20.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:20.105+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:25.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:25.105+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:30.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:30.105+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:35.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:35.106+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:40.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:40.106+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:45.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:45.107+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:50.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:50.108+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:57:53.397 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-09T22:57:53.397+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-09T22:57:55.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:57:55.108+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:00.109+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:05.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:05.109+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:10.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:10.110+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:15.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:15.110+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:20.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:20.111+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:25.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:25.112+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:30.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:30.112+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:35.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:35.113+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:40.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:40.114+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:45.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:45.114+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:50.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:50.115+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:58:55.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:58:55.116+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:00.117+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:05.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:05.117+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:10.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:10.117+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:15.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:15.118+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:20.118+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:25.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:25.119+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-09T22:59:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:30.120+0000 7fe0fc0e8780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T22:59:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:30.120+0000 7fe0fc0e8780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:59:30.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:30.120+0000 7fe0fc0e8780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T22:59:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T22:59:30.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-04-09T22:59:30.142 DEBUG:boto:path=/xryqqk-57/ 2024-04-09T22:59:30.142 DEBUG:boto:auth_path=/xryqqk-57/ 2024-04-09T22:59:30.142 DEBUG:boto:Method: PUT 2024-04-09T22:59:30.142 DEBUG:boto:Path: /xryqqk-57/ 2024-04-09T22:59:30.143 DEBUG:boto:Data: 2024-04-09T22:59:30.143 DEBUG:boto:Headers: {} 2024-04-09T22:59:30.143 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.143 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.143 DEBUG:boto:Params: {} 2024-04-09T22:59:30.143 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T22:59:30.144 DEBUG:boto:Token: None 2024-04-09T22:59:30.144 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/ 2024-04-09T22:59:30.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SM+vEIeJP8I89kbZoWYRGi5quts= 2024-04-09T22:59:30.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SM+vEIeJP8I89kbZoWYRGi5quts=', 'Content-Length': '0'} 2024-04-09T22:59:30.167 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007aebc3a5ad464650-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.168 DEBUG:rgw_multi.tests:created bucket=xryqqk-57 2024-04-09T22:59:30.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:59:30.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:59:30.169 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:59:30.254 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:59:30.255 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:59:30.292 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T22:59:30.332 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T22:59:30.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.332 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:59:30.332 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T22:59:30.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.332 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703570.166272_914.1", 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T22:59:30.166272Z" 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:59:30.333 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T22:59:30.335 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703570.166272_914.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:59:30.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T22:59:30.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:59:30.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T22:59:30.335 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T22:59:30.385 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:59:30.385 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:59:30.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:30.434+0000 7f071baf7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:59:30.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:30.435+0000 7f071baf7780 20 RGW-SYNC:meta: read sync status 2024-04-09T22:59:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.455 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703570.166272_914.1", 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T22:59:30.166272Z", 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:59:30.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T22:59:30.459 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T22:59:30Z" 2024-04-09T22:59:30.460 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T22:59:30.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712703570.166272_914.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T22:59:30.166272Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T22:59:30Z" } 2024-04-09T22:59:30.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703570.166272_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:59:30.166272Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T22:59:30.462 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703570.166272_914.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:59:30.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703570.166272_914.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T22:59:30.462 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T22:59:30.462 DEBUG:boto:path=/xryqqk-57/ 2024-04-09T22:59:30.462 DEBUG:boto:auth_path=/xryqqk-57/ 2024-04-09T22:59:30.462 DEBUG:boto:Method: HEAD 2024-04-09T22:59:30.462 DEBUG:boto:Path: /xryqqk-57/ 2024-04-09T22:59:30.462 DEBUG:boto:Data: 2024-04-09T22:59:30.463 DEBUG:boto:Headers: {} 2024-04-09T22:59:30.463 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.463 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.463 DEBUG:boto:Params: {} 2024-04-09T22:59:30.463 DEBUG:boto:Token: None 2024-04-09T22:59:30.463 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/ 2024-04-09T22:59:30.463 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-09T22:59:30.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-09T22:59:30.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ed19f380d11ff09-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.466 DEBUG:boto:path=/xryqqk-57/a 2024-04-09T22:59:30.466 DEBUG:boto:auth_path=/xryqqk-57/a 2024-04-09T22:59:30.466 DEBUG:boto:Method: PUT 2024-04-09T22:59:30.466 DEBUG:boto:Path: /xryqqk-57/a 2024-04-09T22:59:30.466 DEBUG:boto:Data: 2024-04-09T22:59:30.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:59:30.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.467 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.467 DEBUG:boto:Params: {} 2024-04-09T22:59:30.467 DEBUG:boto:Token: None 2024-04-09T22:59:30.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/a 2024-04-09T22:59:30.467 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dqOnFqwp2XddQJzGEGuBzxGV7h8= 2024-04-09T22:59:30.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dqOnFqwp2XddQJzGEGuBzxGV7h8='} 2024-04-09T22:59:30.476 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe6dc4fb378b7a50-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.476 DEBUG:boto:path=/xryqqk-57/ 2024-04-09T22:59:30.476 DEBUG:boto:auth_path=/xryqqk-57/ 2024-04-09T22:59:30.476 DEBUG:boto:Method: HEAD 2024-04-09T22:59:30.476 DEBUG:boto:Path: /xryqqk-57/ 2024-04-09T22:59:30.476 DEBUG:boto:Data: 2024-04-09T22:59:30.476 DEBUG:boto:Headers: {} 2024-04-09T22:59:30.476 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.476 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.476 DEBUG:boto:Params: {} 2024-04-09T22:59:30.477 DEBUG:boto:Token: None 2024-04-09T22:59:30.477 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/ 2024-04-09T22:59:30.477 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-09T22:59:30.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-09T22:59:30.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8870d7452f0002d-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.479 DEBUG:boto:path=/xryqqk-57/b 2024-04-09T22:59:30.479 DEBUG:boto:auth_path=/xryqqk-57/b 2024-04-09T22:59:30.479 DEBUG:boto:Method: PUT 2024-04-09T22:59:30.479 DEBUG:boto:Path: /xryqqk-57/b 2024-04-09T22:59:30.479 DEBUG:boto:Data: 2024-04-09T22:59:30.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:59:30.480 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.480 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.480 DEBUG:boto:Params: {} 2024-04-09T22:59:30.480 DEBUG:boto:Token: None 2024-04-09T22:59:30.480 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/b 2024-04-09T22:59:30.480 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5HWFV5DZWgq+R802g0ymRCPzNxk= 2024-04-09T22:59:30.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5HWFV5DZWgq+R802g0ymRCPzNxk='} 2024-04-09T22:59:30.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000410e7199b24fac32-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.487 DEBUG:boto:path=/xryqqk-57/ 2024-04-09T22:59:30.487 DEBUG:boto:auth_path=/xryqqk-57/ 2024-04-09T22:59:30.487 DEBUG:boto:Method: HEAD 2024-04-09T22:59:30.487 DEBUG:boto:Path: /xryqqk-57/ 2024-04-09T22:59:30.487 DEBUG:boto:Data: 2024-04-09T22:59:30.488 DEBUG:boto:Headers: {} 2024-04-09T22:59:30.488 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.488 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.488 DEBUG:boto:Params: {} 2024-04-09T22:59:30.488 DEBUG:boto:Token: None 2024-04-09T22:59:30.488 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/ 2024-04-09T22:59:30.488 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-09T22:59:30.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-09T22:59:30.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000555e032fc8d2edd-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.491 DEBUG:boto:path=/xryqqk-57/c 2024-04-09T22:59:30.491 DEBUG:boto:auth_path=/xryqqk-57/c 2024-04-09T22:59:30.491 DEBUG:boto:Method: PUT 2024-04-09T22:59:30.491 DEBUG:boto:Path: /xryqqk-57/c 2024-04-09T22:59:30.492 DEBUG:boto:Data: 2024-04-09T22:59:30.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:59:30.492 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.492 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.492 DEBUG:boto:Params: {} 2024-04-09T22:59:30.492 DEBUG:boto:Token: None 2024-04-09T22:59:30.492 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/c 2024-04-09T22:59:30.492 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:opJXVWuK8D8mqAuPaBjSBmJ6hXU= 2024-04-09T22:59:30.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:opJXVWuK8D8mqAuPaBjSBmJ6hXU='} 2024-04-09T22:59:30.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069a04d861003af52-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.539 DEBUG:boto:path=/xryqqk-57/ 2024-04-09T22:59:30.539 DEBUG:boto:auth_path=/xryqqk-57/ 2024-04-09T22:59:30.539 DEBUG:boto:Method: HEAD 2024-04-09T22:59:30.539 DEBUG:boto:Path: /xryqqk-57/ 2024-04-09T22:59:30.539 DEBUG:boto:Data: 2024-04-09T22:59:30.539 DEBUG:boto:Headers: {} 2024-04-09T22:59:30.539 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.539 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.539 DEBUG:boto:Params: {} 2024-04-09T22:59:30.540 DEBUG:boto:Token: None 2024-04-09T22:59:30.540 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/ 2024-04-09T22:59:30.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-09T22:59:30.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-09T22:59:30.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033ea05df42a7c9a7-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.542 DEBUG:boto:path=/xryqqk-57/d 2024-04-09T22:59:30.542 DEBUG:boto:auth_path=/xryqqk-57/d 2024-04-09T22:59:30.543 DEBUG:boto:Method: PUT 2024-04-09T22:59:30.543 DEBUG:boto:Path: /xryqqk-57/d 2024-04-09T22:59:30.543 DEBUG:boto:Data: 2024-04-09T22:59:30.543 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T22:59:30.543 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T22:59:30.543 DEBUG:boto:Port: 8000 2024-04-09T22:59:30.543 DEBUG:boto:Params: {} 2024-04-09T22:59:30.543 DEBUG:boto:Token: None 2024-04-09T22:59:30.543 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 22:59:30 GMT /xryqqk-57/d 2024-04-09T22:59:30.544 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XkeaYmBVdS9Y8MY/HfpJPldSDvw= 2024-04-09T22:59:30.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XkeaYmBVdS9Y8MY/HfpJPldSDvw='} 2024-04-09T22:59:30.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000807f10672a21c21e-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T22:59:30.552 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-57 2024-04-09T22:59:30.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:59:30.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:59:30.553 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:59:30.611 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:59:30.611 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-57 2024-04-09T22:59:30.701 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-57 2024-04-09T22:59:30.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:59:30.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T22:59:30.702 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T22:59:30.759 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T22:59:30.759 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T22:59:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:30.797+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T22:59:35.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:35.798+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T22:59:40.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:40.798+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T22:59:45.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:45.799+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T22:59:50.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:50.799+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T22:59:55.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T22:59:55.800+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:00.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:00.800+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:05.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:05.801+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:10.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:10.802+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:15.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:15.802+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:20.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:20.803+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:25.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:25.804+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:30.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:30.804+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:35.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:35.805+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:40.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:40.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:45.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:45.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:50.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:50.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:00:55.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:00:55.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:00.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:00.807+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:05.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:05.807+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:10.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:10.808+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:15.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:15.808+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:20.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:20.809+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:25.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:25.810+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:30.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:30.810+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:35.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:35.811+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:40.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:40.812+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:45.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:45.812+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:50.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:50.813+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:01:55.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:01:55.813+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:00.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:00.814+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:05.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:05.815+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:10.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:10.815+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:15.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:15.816+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:20.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:20.817+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:25.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:25.817+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:30.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:30.817+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:35.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:35.818+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:40.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:40.818+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:45.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:45.819+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:50.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:50.819+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:02:55.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:02:55.820+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:00.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:00.821+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:05.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:05.821+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:10.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:10.822+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:15.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:15.822+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:20.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:20.823+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:25.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:25.823+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:30.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:30.824+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:35.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:35.825+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:40.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:40.825+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:45.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:45.826+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:50.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:50.827+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:03:55.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:03:55.827+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:04:00.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:00.828+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:04:05.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:05.829+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:10.829+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:04:15.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:15.829+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:04:20.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:20.830+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:25.830+0000 7f0c36861780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:25.830+0000 7f0c36861780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:25.830+0000 7f0c36861780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:04:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:04:25.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... FAIL 2024-04-09T23:04:25.854 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:25.854 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:25.854 DEBUG:boto:Method: PUT 2024-04-09T23:04:25.854 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:25.854 DEBUG:boto:Data: 2024-04-09T23:04:25.854 DEBUG:boto:Headers: {} 2024-04-09T23:04:25.854 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:25.854 DEBUG:boto:Port: 8000 2024-04-09T23:04:25.855 DEBUG:boto:Params: {} 2024-04-09T23:04:25.855 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:04:25.855 DEBUG:boto:Token: None 2024-04-09T23:04:25.855 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:04:25 GMT /xryqqk-58/ 2024-04-09T23:04:25.856 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y7ny92BpUdFLHdDHMrqeuaNaMdU= 2024-04-09T23:04:25.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:25 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y7ny92BpUdFLHdDHMrqeuaNaMdU=', 'Content-Length': '0'} 2024-04-09T23:04:25.882 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009884e99599f56db3-006615c979-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:25.883 DEBUG:rgw_multi.tests:created bucket=xryqqk-58 2024-04-09T23:04:25.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:04:25.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:04:25.883 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:04:25.936 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:04:25.936 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:04:25.980 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:04:26.008 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703865.880236_928.1", 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:04:25.880236Z" 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:26.010 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:04:26.011 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:04:26.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:04:26.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:04:26.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:04:26.012 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:04:26.064 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:04:26.064 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:04:26.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:26.117+0000 7fadaeda8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:04:26.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:26.118+0000 7fadaeda8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:04:26.138 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:04:26.138 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:04:26.138 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:04:26.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:04:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:26.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703865.870934_925.1", 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:04:25.870934Z", 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:04:26.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:26.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:04:26Z" 2024-04-09T23:04:26.144 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:04:26.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712703865.870934_925.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:04:25.870934Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:04:26Z" } 2024-04-09T23:04:26.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703865.870934_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:04:25.870934Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:04:26.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:04:26.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.870934_925.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:04:26.146 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712703865.880236_928.1 target=1_1712703865.870934_925.1 2024-04-09T23:04:31.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:04:31.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:04:31.148 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:04:31.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:04:31.207 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:04:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:31.252+0000 7fbf48537780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:04:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:31.253+0000 7fbf48537780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703865.880236_928.1", 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:04:25.880236Z", 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:04:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:04:31Z" 2024-04-09T23:04:31.280 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:04:31.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712703865.880236_928.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:04:25.880236Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:04:31Z" } 2024-04-09T23:04:31.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703865.880236_928.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:04:25.880236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:04:31.282 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:04:31.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:04:31.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:04:31.282 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-09T23:04:31.282 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T23:04:31.283 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:04:31.283+0000 7f8b978c8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 49440) UID: 0 2024-04-09T23:04:31.283 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:04:31.283+0000 7f8b9ca9d7c0 -1 shutting down 2024-04-09T23:04:35.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-09T23:04:37.384 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-09T23:04:37.384 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.384 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.384 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.385 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.385 DEBUG:boto:Data: 2024-04-09T23:04:37.385 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.385 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.385 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.385 DEBUG:boto:Params: {} 2024-04-09T23:04:37.385 DEBUG:boto:Token: None 2024-04-09T23:04:37.385 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.386 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aba5d5d57a69510-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.389 DEBUG:boto:path=/xryqqk-58/a 2024-04-09T23:04:37.389 DEBUG:boto:auth_path=/xryqqk-58/a 2024-04-09T23:04:37.389 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.389 DEBUG:boto:Path: /xryqqk-58/a 2024-04-09T23:04:37.389 DEBUG:boto:Data: 2024-04-09T23:04:37.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.390 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.390 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.390 DEBUG:boto:Params: {} 2024-04-09T23:04:37.390 DEBUG:boto:Token: None 2024-04-09T23:04:37.390 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/a 2024-04-09T23:04:37.390 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IWfY3eLOntdWbL5N2GRseQc5vLY= 2024-04-09T23:04:37.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IWfY3eLOntdWbL5N2GRseQc5vLY='} 2024-04-09T23:04:37.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056b12ffdbac838f1-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.398 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.398 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.398 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.399 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.399 DEBUG:boto:Data: 2024-04-09T23:04:37.399 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.399 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.399 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.399 DEBUG:boto:Params: {} 2024-04-09T23:04:37.399 DEBUG:boto:Token: None 2024-04-09T23:04:37.399 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.399 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.402 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083802fb31b78b52b-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.402 DEBUG:boto:path=/xryqqk-58/b 2024-04-09T23:04:37.402 DEBUG:boto:auth_path=/xryqqk-58/b 2024-04-09T23:04:37.402 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.402 DEBUG:boto:Path: /xryqqk-58/b 2024-04-09T23:04:37.403 DEBUG:boto:Data: 2024-04-09T23:04:37.403 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.403 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.403 DEBUG:boto:Params: {} 2024-04-09T23:04:37.403 DEBUG:boto:Token: None 2024-04-09T23:04:37.403 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/b 2024-04-09T23:04:37.403 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rszmoV51IqhmBCnlB/M6Ph1aFbs= 2024-04-09T23:04:37.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rszmoV51IqhmBCnlB/M6Ph1aFbs='} 2024-04-09T23:04:37.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000450fc04c85f6090c-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.409 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.410 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.410 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.410 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.410 DEBUG:boto:Data: 2024-04-09T23:04:37.410 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.410 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.410 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.410 DEBUG:boto:Params: {} 2024-04-09T23:04:37.410 DEBUG:boto:Token: None 2024-04-09T23:04:37.411 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.411 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a893907e4152df-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.413 DEBUG:boto:path=/xryqqk-58/c 2024-04-09T23:04:37.413 DEBUG:boto:auth_path=/xryqqk-58/c 2024-04-09T23:04:37.413 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.414 DEBUG:boto:Path: /xryqqk-58/c 2024-04-09T23:04:37.414 DEBUG:boto:Data: 2024-04-09T23:04:37.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.414 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.414 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.414 DEBUG:boto:Params: {} 2024-04-09T23:04:37.414 DEBUG:boto:Token: None 2024-04-09T23:04:37.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/c 2024-04-09T23:04:37.414 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vIvcbcP+DRTWRJpNrwytAHNWrqI= 2024-04-09T23:04:37.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vIvcbcP+DRTWRJpNrwytAHNWrqI='} 2024-04-09T23:04:37.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af87ea30ca238667-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.422 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.422 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.423 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.423 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.423 DEBUG:boto:Data: 2024-04-09T23:04:37.423 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.423 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.423 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.423 DEBUG:boto:Params: {} 2024-04-09T23:04:37.423 DEBUG:boto:Token: None 2024-04-09T23:04:37.423 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.423 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000469be5c22fa3ff19-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.426 DEBUG:boto:path=/xryqqk-58/d 2024-04-09T23:04:37.426 DEBUG:boto:auth_path=/xryqqk-58/d 2024-04-09T23:04:37.426 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.427 DEBUG:boto:Path: /xryqqk-58/d 2024-04-09T23:04:37.427 DEBUG:boto:Data: 2024-04-09T23:04:37.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.427 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.427 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.427 DEBUG:boto:Params: {} 2024-04-09T23:04:37.427 DEBUG:boto:Token: None 2024-04-09T23:04:37.427 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/d 2024-04-09T23:04:37.427 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4H1z4gVrVaZ6B83iboWhYSWM53U= 2024-04-09T23:04:37.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4H1z4gVrVaZ6B83iboWhYSWM53U='} 2024-04-09T23:04:37.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d0c295c9eb0f884-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'xryqqk-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:04:37.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'xryqqk-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:04:37.475 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket xryqqk-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:04:37.528 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:04:37.528 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:04:37.604 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:04:37.604 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-58 2024-04-09T23:04:37.608 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 4 2024-04-09T23:04:37.658 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.658 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.658 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.658 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.658 DEBUG:boto:Data: 2024-04-09T23:04:37.658 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.658 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.659 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.659 DEBUG:boto:Params: {} 2024-04-09T23:04:37.659 DEBUG:boto:Token: None 2024-04-09T23:04:37.659 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.659 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.661 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000978247dff672cdb3-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.661 DEBUG:boto:path=/xryqqk-58/e 2024-04-09T23:04:37.661 DEBUG:boto:auth_path=/xryqqk-58/e 2024-04-09T23:04:37.662 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.662 DEBUG:boto:Path: /xryqqk-58/e 2024-04-09T23:04:37.662 DEBUG:boto:Data: 2024-04-09T23:04:37.662 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.662 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.662 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.662 DEBUG:boto:Params: {} 2024-04-09T23:04:37.662 DEBUG:boto:Token: None 2024-04-09T23:04:37.662 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/e 2024-04-09T23:04:37.663 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hpfZFfSImcTMXnYq91Pj7wZFMPc= 2024-04-09T23:04:37.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hpfZFfSImcTMXnYq91Pj7wZFMPc='} 2024-04-09T23:04:37.668 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001681e049f7510c20-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.668 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.668 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.668 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.668 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.669 DEBUG:boto:Data: 2024-04-09T23:04:37.669 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.669 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.669 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.669 DEBUG:boto:Params: {} 2024-04-09T23:04:37.669 DEBUG:boto:Token: None 2024-04-09T23:04:37.669 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.669 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000645333ae9380eb17-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.672 DEBUG:boto:path=/xryqqk-58/f 2024-04-09T23:04:37.672 DEBUG:boto:auth_path=/xryqqk-58/f 2024-04-09T23:04:37.672 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.672 DEBUG:boto:Path: /xryqqk-58/f 2024-04-09T23:04:37.672 DEBUG:boto:Data: 2024-04-09T23:04:37.673 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.673 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.673 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.673 DEBUG:boto:Params: {} 2024-04-09T23:04:37.673 DEBUG:boto:Token: None 2024-04-09T23:04:37.673 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/f 2024-04-09T23:04:37.673 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HnnKczJmRNBZ0qUyqYIdAPKj/d4= 2024-04-09T23:04:37.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HnnKczJmRNBZ0qUyqYIdAPKj/d4='} 2024-04-09T23:04:37.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000202d12105d460cec-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.679 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.679 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.679 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.679 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.679 DEBUG:boto:Data: 2024-04-09T23:04:37.679 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.679 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.679 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.680 DEBUG:boto:Params: {} 2024-04-09T23:04:37.680 DEBUG:boto:Token: None 2024-04-09T23:04:37.680 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.680 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031e924043ad36d4b-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.682 DEBUG:boto:path=/xryqqk-58/g 2024-04-09T23:04:37.682 DEBUG:boto:auth_path=/xryqqk-58/g 2024-04-09T23:04:37.682 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.682 DEBUG:boto:Path: /xryqqk-58/g 2024-04-09T23:04:37.682 DEBUG:boto:Data: 2024-04-09T23:04:37.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.682 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.682 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.683 DEBUG:boto:Params: {} 2024-04-09T23:04:37.683 DEBUG:boto:Token: None 2024-04-09T23:04:37.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/g 2024-04-09T23:04:37.683 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ITA9sGl8xDhbVCnj0+9uWBMg75Q= 2024-04-09T23:04:37.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ITA9sGl8xDhbVCnj0+9uWBMg75Q='} 2024-04-09T23:04:37.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fadb3ad8b2a69a1-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.727 DEBUG:boto:path=/xryqqk-58/ 2024-04-09T23:04:37.727 DEBUG:boto:auth_path=/xryqqk-58/ 2024-04-09T23:04:37.727 DEBUG:boto:Method: HEAD 2024-04-09T23:04:37.727 DEBUG:boto:Path: /xryqqk-58/ 2024-04-09T23:04:37.727 DEBUG:boto:Data: 2024-04-09T23:04:37.727 DEBUG:boto:Headers: {} 2024-04-09T23:04:37.727 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.727 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.727 DEBUG:boto:Params: {} 2024-04-09T23:04:37.728 DEBUG:boto:Token: None 2024-04-09T23:04:37.728 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/ 2024-04-09T23:04:37.728 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-09T23:04:37.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-09T23:04:37.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c43b615b1c1fe21-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.730 DEBUG:boto:path=/xryqqk-58/h 2024-04-09T23:04:37.730 DEBUG:boto:auth_path=/xryqqk-58/h 2024-04-09T23:04:37.730 DEBUG:boto:Method: PUT 2024-04-09T23:04:37.730 DEBUG:boto:Path: /xryqqk-58/h 2024-04-09T23:04:37.730 DEBUG:boto:Data: 2024-04-09T23:04:37.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:04:37.731 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:04:37.731 DEBUG:boto:Port: 8000 2024-04-09T23:04:37.731 DEBUG:boto:Params: {} 2024-04-09T23:04:37.731 DEBUG:boto:Token: None 2024-04-09T23:04:37.731 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:04:37 GMT /xryqqk-58/h 2024-04-09T23:04:37.731 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9pvW5QBRcdJ3Lb4tKDRESnFIUWs= 2024-04-09T23:04:37.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9pvW5QBRcdJ3Lb4tKDRESnFIUWs='} 2024-04-09T23:04:37.735 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bafc4c0127dcf9b-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:04:37.735 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-09T23:04:37.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-09T23:04:37.738 INFO:tasks.rgw.c1.client.1:Started 2024-04-09T23:04:37.738 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T23:04:37.786 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T23:04:37.786 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T23:04:37.788 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T23:04:37.788 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-09T23:04:38.790 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T23:04:38.812 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T23:04:38.812 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T23:04:38.813 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T23:04:38.814 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T23:04:38.814 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-58 2024-04-09T23:04:38.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:04:38.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:04:38.815 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:04:38.911 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:04:38.911 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:04:38.964 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-58 2024-04-09T23:04:38.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-58 2024-04-09T23:04:38.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:04:38.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:04:38.987 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:04:39.043 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:04:39.043 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:04:39.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:39.081+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:04:41.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-09T23:04:44.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:44.081+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:04:49.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:49.082+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:04:54.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:54.082+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:04:59.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:04:59.083+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:04.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:04.083+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:09.084+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:14.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:14.085+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:19.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:19.085+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:24.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:24.086+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:29.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:29.086+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:34.086+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:39.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:39.087+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:44.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:44.089+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:49.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:49.088+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:54.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:54.089+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:05:59.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:05:59.090+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:04.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:04.090+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:09.091+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:14.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:14.091+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:19.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:19.092+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:24.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:24.092+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:29.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:29.093+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:34.094+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:39.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:39.094+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:44.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:44.095+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:49.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:49.095+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:54.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:54.096+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:06:59.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:06:59.096+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:04.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:04.097+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:09.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:09.097+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:14.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:14.098+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:19.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:19.098+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:24.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:24.100+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:29.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:29.100+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:34.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:34.100+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:39.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:39.101+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:44.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:44.102+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:49.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:49.102+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:54.103+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:07:59.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:07:59.103+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:04.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:04.104+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:09.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:09.105+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:14.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:14.105+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:19.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:19.106+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:24.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:24.106+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:29.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:29.107+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:34.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:34.108+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:39.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:39.108+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:44.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:44.109+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:49.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:49.109+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:54.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:54.111+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:08:59.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:08:59.111+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:04.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:04.112+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:09.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:09.113+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:14.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:14.113+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:19.113+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:24.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:24.114+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:29.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:29.114+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-09T23:09:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.115+0000 7f8d3a007780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:09:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.115+0000 7f8d3a007780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:09:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.115+0000 7f8d3a007780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:09:34.133 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:09:34.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... FAIL 2024-04-09T23:09:34.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.137 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:09:34.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:34.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:34.258 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:09:34.272 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:09:34.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.272 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:09:34.272 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T23:09:34.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703877.617586_941.1", 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:04:37.617586Z" 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.274 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:09:34.274 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:09:34.274 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.274 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:09:34.275 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703877.617586_941.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:09:34.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:09:34.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.276 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:09:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:34.326 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:34.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.370+0000 7f244cd19780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:09:34.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.371+0000 7f244cd19780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:09:34.394 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.395 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:09:34.396 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.397 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712703877.617586_941.1", 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:04:37.617586Z", 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.398 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:09:34.399 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:09:34.400 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:09:34.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.400 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:09:34Z" 2024-04-09T23:09:34.400 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:09:34.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712703877.617586_941.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:04:37.617586Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:09:34Z" } 2024-04-09T23:09:34.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703877.617586_941.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:04:37.617586Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:09:34.401 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703877.617586_941.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:09:34.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703877.617586_941.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:09:34.401 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:09:34.401 DEBUG:boto:path=/ 2024-04-09T23:09:34.402 DEBUG:boto:auth_path=/ 2024-04-09T23:09:34.402 DEBUG:boto:Method: GET 2024-04-09T23:09:34.402 DEBUG:boto:Path: / 2024-04-09T23:09:34.402 DEBUG:boto:Data: 2024-04-09T23:09:34.402 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.402 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.402 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.402 DEBUG:boto:Params: {} 2024-04-09T23:09:34.403 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:09:34.403 DEBUG:boto:Token: None 2024-04-09T23:09:34.403 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:09:34 GMT / 2024-04-09T23:09:34.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7Dku/lsKWO+35ANKs2UsapNTLts= 2024-04-09T23:09:34.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Dku/lsKWO+35ANKs2UsapNTLts=', 'Content-Length': '0'} 2024-04-09T23:09:34.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092b73ac02c099d88-006615caae-1b6d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.411 DEBUG:boto:path=/ 2024-04-09T23:09:34.411 DEBUG:boto:auth_path=/ 2024-04-09T23:09:34.411 DEBUG:boto:Method: GET 2024-04-09T23:09:34.411 DEBUG:boto:Path: / 2024-04-09T23:09:34.411 DEBUG:boto:Data: 2024-04-09T23:09:34.411 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.411 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:09:34.411 DEBUG:boto:Port: 8001 2024-04-09T23:09:34.412 DEBUG:boto:Params: {} 2024-04-09T23:09:34.412 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:09:34.412 DEBUG:boto:Token: None 2024-04-09T23:09:34.412 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:09:34 GMT / 2024-04-09T23:09:34.412 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7Dku/lsKWO+35ANKs2UsapNTLts= 2024-04-09T23:09:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Dku/lsKWO+35ANKs2UsapNTLts=', 'Content-Length': '0'} 2024-04-09T23:09:34.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b7d41866abe1868-006615caae-1fc6-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-04-09T23:09:34.419 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.419 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.419 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.419 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.419 DEBUG:boto:Data: 2024-04-09T23:09:34.419 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.419 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.419 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.419 DEBUG:boto:Params: {} 2024-04-09T23:09:34.420 DEBUG:boto:Token: None 2024-04-09T23:09:34.420 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.420 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:x6jYdnOWEsi5V6O1kEQmgr31AmM= 2024-04-09T23:09:34.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x6jYdnOWEsi5V6O1kEQmgr31AmM=', 'Content-Length': '0'} 2024-04-09T23:09:34.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed3284f86aa95ec5-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.446 DEBUG:rgw_multi.tests:created bucket=xryqqk-59 2024-04-09T23:09:34.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.446 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:09:34.496 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:34.496 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:34.549 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:09:34.564 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:09:34.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.564 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:09:34.564 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T23:09:34.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704174.442514_998.1", 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:09:34.442514Z" 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:09:34.565 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:09:34.566 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.566 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:09:34.568 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704174.442514_998.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:09:34.568 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:09:34.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.568 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:09:34.622 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:34.622 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.660+0000 7ff870192780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:09:34.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:34.661+0000 7ff870192780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:09:34.683 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:09:34.683 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.684 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.685 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:09:34.686 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704174.442514_998.1", 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:09:34.442514Z", 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:09:34.687 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:09:34.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:09:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:09:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:09:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:09:34.689 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:09:34Z" 2024-04-09T23:09:34.689 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:09:34.690 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712704174.442514_998.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:09:34.442514Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:09:34Z" } 2024-04-09T23:09:34.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704174.442514_998.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:09:34.442514Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:09:34.690 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704174.442514_998.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:09:34.690 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704174.442514_998.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:09:34.690 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:09:34.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:09:34.691 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:09:34.742 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:34.742 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "objs": [ 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000001", 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000005", 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000012", 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000004", 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000011", 2024-04-09T23:09:34.823 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000000", 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000010", 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000003", 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000013", 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000002", 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout: "obj00000000000000000006" 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:09:34.824 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:09:34.825 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.825 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.825 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.825 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.826 DEBUG:boto:Data: 2024-04-09T23:09:34.826 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.826 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.826 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.826 DEBUG:boto:Params: {} 2024-04-09T23:09:34.826 DEBUG:boto:Token: None 2024-04-09T23:09:34.826 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.826 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000323abcd6139ccead-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.829 DEBUG:boto:path=/xryqqk-59/obj00000000000000000001 2024-04-09T23:09:34.829 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000001 2024-04-09T23:09:34.829 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.829 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000001 2024-04-09T23:09:34.829 DEBUG:boto:Data: 2024-04-09T23:09:34.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.829 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.830 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.830 DEBUG:boto:Params: {} 2024-04-09T23:09:34.830 DEBUG:boto:Token: None 2024-04-09T23:09:34.830 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000001 2024-04-09T23:09:34.830 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:suTJFYHEYYrHQF20I9S2AvgORdw= 2024-04-09T23:09:34.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:suTJFYHEYYrHQF20I9S2AvgORdw='} 2024-04-09T23:09:34.836 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ba891640121f9fd-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.837 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.837 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.837 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.837 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.837 DEBUG:boto:Data: 2024-04-09T23:09:34.837 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.837 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.838 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.838 DEBUG:boto:Params: {} 2024-04-09T23:09:34.838 DEBUG:boto:Token: None 2024-04-09T23:09:34.838 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.838 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034a8ce7d1a766071-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.840 DEBUG:boto:path=/xryqqk-59/obj00000000000000000012 2024-04-09T23:09:34.841 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000012 2024-04-09T23:09:34.841 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.842 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000012 2024-04-09T23:09:34.842 DEBUG:boto:Data: 2024-04-09T23:09:34.842 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.842 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.842 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.842 DEBUG:boto:Params: {} 2024-04-09T23:09:34.842 DEBUG:boto:Token: None 2024-04-09T23:09:34.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000012 2024-04-09T23:09:34.843 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xgUcG4goJJLUHkTLLoAailK8a8Y= 2024-04-09T23:09:34.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xgUcG4goJJLUHkTLLoAailK8a8Y='} 2024-04-09T23:09:34.852 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ba8bc5997294082-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.852 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.852 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.853 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.853 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.853 DEBUG:boto:Data: 2024-04-09T23:09:34.853 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.853 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.853 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.853 DEBUG:boto:Params: {} 2024-04-09T23:09:34.853 DEBUG:boto:Token: None 2024-04-09T23:09:34.853 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.853 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000425f8c461467a8e9-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.856 DEBUG:boto:path=/xryqqk-59/obj00000000000000000003 2024-04-09T23:09:34.856 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000003 2024-04-09T23:09:34.856 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.856 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000003 2024-04-09T23:09:34.856 DEBUG:boto:Data: 2024-04-09T23:09:34.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.856 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.856 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.856 DEBUG:boto:Params: {} 2024-04-09T23:09:34.857 DEBUG:boto:Token: None 2024-04-09T23:09:34.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000003 2024-04-09T23:09:34.857 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YM0iWAeyhhLf3Hz1ZfBvn0hkZqc= 2024-04-09T23:09:34.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YM0iWAeyhhLf3Hz1ZfBvn0hkZqc='} 2024-04-09T23:09:34.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041f8e693eff16b66-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.899 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.899 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.899 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.900 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.900 DEBUG:boto:Data: 2024-04-09T23:09:34.900 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.900 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.900 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.900 DEBUG:boto:Params: {} 2024-04-09T23:09:34.900 DEBUG:boto:Token: None 2024-04-09T23:09:34.901 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.901 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f206023508fbf67-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.903 DEBUG:boto:path=/xryqqk-59/obj00000000000000000013 2024-04-09T23:09:34.903 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000013 2024-04-09T23:09:34.903 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.903 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000013 2024-04-09T23:09:34.903 DEBUG:boto:Data: 2024-04-09T23:09:34.904 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.904 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.904 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.904 DEBUG:boto:Params: {} 2024-04-09T23:09:34.904 DEBUG:boto:Token: None 2024-04-09T23:09:34.904 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000013 2024-04-09T23:09:34.904 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YN2py/4E5deOkX8sTC3CtyLaZus= 2024-04-09T23:09:34.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YN2py/4E5deOkX8sTC3CtyLaZus='} 2024-04-09T23:09:34.916 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4b52410cdd39467-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.916 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.916 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.916 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.916 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.917 DEBUG:boto:Data: 2024-04-09T23:09:34.917 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.917 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.917 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.917 DEBUG:boto:Params: {} 2024-04-09T23:09:34.917 DEBUG:boto:Token: None 2024-04-09T23:09:34.917 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.917 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.919 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc2dfc43ed6485de-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.920 DEBUG:boto:path=/xryqqk-59/obj00000000000000000006 2024-04-09T23:09:34.920 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000006 2024-04-09T23:09:34.920 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.920 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000006 2024-04-09T23:09:34.920 DEBUG:boto:Data: 2024-04-09T23:09:34.920 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.920 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.920 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.920 DEBUG:boto:Params: {} 2024-04-09T23:09:34.921 DEBUG:boto:Token: None 2024-04-09T23:09:34.921 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000006 2024-04-09T23:09:34.921 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HGURoYFbBbQJ2L3//gZ/HVK9hcI= 2024-04-09T23:09:34.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HGURoYFbBbQJ2L3//gZ/HVK9hcI='} 2024-04-09T23:09:34.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000881ea156056bfc06-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.967 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.967 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.967 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.967 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.967 DEBUG:boto:Data: 2024-04-09T23:09:34.967 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.967 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.967 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.968 DEBUG:boto:Params: {} 2024-04-09T23:09:34.968 DEBUG:boto:Token: None 2024-04-09T23:09:34.968 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.968 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.970 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb4b063d27e465a5-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.970 DEBUG:boto:path=/xryqqk-59/obj00000000000000000000 2024-04-09T23:09:34.971 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000000 2024-04-09T23:09:34.971 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.971 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000000 2024-04-09T23:09:34.971 DEBUG:boto:Data: 2024-04-09T23:09:34.971 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.971 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.971 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.971 DEBUG:boto:Params: {} 2024-04-09T23:09:34.971 DEBUG:boto:Token: None 2024-04-09T23:09:34.971 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000000 2024-04-09T23:09:34.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QbwL3huOV9X5VQTtFEoem/u+13I= 2024-04-09T23:09:34.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QbwL3huOV9X5VQTtFEoem/u+13I='} 2024-04-09T23:09:34.981 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e70b2a777ff92c2b-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.981 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.981 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.981 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.981 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.981 DEBUG:boto:Data: 2024-04-09T23:09:34.981 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.981 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.982 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.982 DEBUG:boto:Params: {} 2024-04-09T23:09:34.982 DEBUG:boto:Token: None 2024-04-09T23:09:34.982 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.982 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f5fdfbf0135bba1-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.984 DEBUG:boto:path=/xryqqk-59/obj00000000000000000010 2024-04-09T23:09:34.984 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000010 2024-04-09T23:09:34.984 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.985 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000010 2024-04-09T23:09:34.985 DEBUG:boto:Data: 2024-04-09T23:09:34.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.985 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.985 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.985 DEBUG:boto:Params: {} 2024-04-09T23:09:34.985 DEBUG:boto:Token: None 2024-04-09T23:09:34.985 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000010 2024-04-09T23:09:34.985 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8BLlRr0Aqkzq0LcKFls6NoZn1h0= 2024-04-09T23:09:34.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8BLlRr0Aqkzq0LcKFls6NoZn1h0='} 2024-04-09T23:09:34.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020133f6a9f04787b-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.994 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:34.995 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:34.995 DEBUG:boto:Method: HEAD 2024-04-09T23:09:34.995 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:34.995 DEBUG:boto:Data: 2024-04-09T23:09:34.995 DEBUG:boto:Headers: {} 2024-04-09T23:09:34.995 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.995 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.995 DEBUG:boto:Params: {} 2024-04-09T23:09:34.995 DEBUG:boto:Token: None 2024-04-09T23:09:34.995 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/ 2024-04-09T23:09:34.996 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-09T23:09:34.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-09T23:09:34.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091497ce79a2af1fc-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:34.998 DEBUG:boto:path=/xryqqk-59/obj00000000000000000005 2024-04-09T23:09:34.998 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000005 2024-04-09T23:09:34.998 DEBUG:boto:Method: PUT 2024-04-09T23:09:34.998 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000005 2024-04-09T23:09:34.998 DEBUG:boto:Data: 2024-04-09T23:09:34.998 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:34.998 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:34.998 DEBUG:boto:Port: 8000 2024-04-09T23:09:34.998 DEBUG:boto:Params: {} 2024-04-09T23:09:34.999 DEBUG:boto:Token: None 2024-04-09T23:09:34.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:34 GMT /xryqqk-59/obj00000000000000000005 2024-04-09T23:09:34.999 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mPZrjFMlynhfKiwQ0CY6SLqnTng= 2024-04-09T23:09:34.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mPZrjFMlynhfKiwQ0CY6SLqnTng='} 2024-04-09T23:09:35.043 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093c9e93f0abb0c58-006615caaf-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:35.043 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:35.043 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:35.043 DEBUG:boto:Method: HEAD 2024-04-09T23:09:35.043 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:35.043 DEBUG:boto:Data: 2024-04-09T23:09:35.043 DEBUG:boto:Headers: {} 2024-04-09T23:09:35.043 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:35.043 DEBUG:boto:Port: 8000 2024-04-09T23:09:35.044 DEBUG:boto:Params: {} 2024-04-09T23:09:35.044 DEBUG:boto:Token: None 2024-04-09T23:09:35.044 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:35 GMT /xryqqk-59/ 2024-04-09T23:09:35.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM= 2024-04-09T23:09:35.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM=', 'Content-Length': '0'} 2024-04-09T23:09:35.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4c3764ec60e6d09-006615caaf-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:35.047 DEBUG:boto:path=/xryqqk-59/obj00000000000000000004 2024-04-09T23:09:35.047 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000004 2024-04-09T23:09:35.047 DEBUG:boto:Method: PUT 2024-04-09T23:09:35.047 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000004 2024-04-09T23:09:35.047 DEBUG:boto:Data: 2024-04-09T23:09:35.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:35.047 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:35.047 DEBUG:boto:Port: 8000 2024-04-09T23:09:35.047 DEBUG:boto:Params: {} 2024-04-09T23:09:35.048 DEBUG:boto:Token: None 2024-04-09T23:09:35.048 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:35 GMT /xryqqk-59/obj00000000000000000004 2024-04-09T23:09:35.048 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:BTa4LKP+jL6Cpyl+hA+0qt8vqRU= 2024-04-09T23:09:35.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BTa4LKP+jL6Cpyl+hA+0qt8vqRU='} 2024-04-09T23:09:35.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9f9c4c7b1e857d4-006615caaf-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:35.056 DEBUG:boto:path=/xryqqk-59/ 2024-04-09T23:09:35.056 DEBUG:boto:auth_path=/xryqqk-59/ 2024-04-09T23:09:35.056 DEBUG:boto:Method: HEAD 2024-04-09T23:09:35.056 DEBUG:boto:Path: /xryqqk-59/ 2024-04-09T23:09:35.057 DEBUG:boto:Data: 2024-04-09T23:09:35.057 DEBUG:boto:Headers: {} 2024-04-09T23:09:35.057 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:35.057 DEBUG:boto:Port: 8000 2024-04-09T23:09:35.057 DEBUG:boto:Params: {} 2024-04-09T23:09:35.057 DEBUG:boto:Token: None 2024-04-09T23:09:35.057 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:09:35 GMT /xryqqk-59/ 2024-04-09T23:09:35.057 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM= 2024-04-09T23:09:35.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM=', 'Content-Length': '0'} 2024-04-09T23:09:35.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf1e5aa04729251f-006615caaf-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:35.059 DEBUG:boto:path=/xryqqk-59/obj00000000000000000002 2024-04-09T23:09:35.060 DEBUG:boto:auth_path=/xryqqk-59/obj00000000000000000002 2024-04-09T23:09:35.060 DEBUG:boto:Method: PUT 2024-04-09T23:09:35.060 DEBUG:boto:Path: /xryqqk-59/obj00000000000000000002 2024-04-09T23:09:35.060 DEBUG:boto:Data: 2024-04-09T23:09:35.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:09:35.060 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:09:35.060 DEBUG:boto:Port: 8000 2024-04-09T23:09:35.061 DEBUG:boto:Params: {} 2024-04-09T23:09:35.061 DEBUG:boto:Token: None 2024-04-09T23:09:35.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:09:35 GMT /xryqqk-59/obj00000000000000000002 2024-04-09T23:09:35.061 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gGgn44Gmzo/Ywxfn9jnS+gtrrHQ= 2024-04-09T23:09:35.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gGgn44Gmzo/Ywxfn9jnS+gtrrHQ='} 2024-04-09T23:09:35.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000419d076e5232b9b1-006615caaf-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:09:35.111 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-59 2024-04-09T23:09:35.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:09:35.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:09:35.111 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:09:35.169 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:35.169 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:35.210 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-59 2024-04-09T23:09:35.224 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-59 2024-04-09T23:09:35.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:09:35.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:09:35.225 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:09:35.282 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:09:35.282 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:09:35.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:35.321+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:09:40.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:40.322+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:09:45.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:45.323+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:09:50.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:50.323+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:09:55.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:09:55.323+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:00.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:00.324+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:05.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:05.325+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:10.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:10.325+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:15.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:15.325+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:20.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:20.326+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:25.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:25.327+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:30.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:30.327+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:35.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:35.327+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:40.328+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:45.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:45.329+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:50.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:50.330+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:10:55.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:10:55.329+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:00.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:00.330+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:05.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:05.331+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:10.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:10.331+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:15.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:15.332+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:20.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:20.333+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:25.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:25.333+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:30.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:30.334+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:35.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:35.334+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:40.335+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:45.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:45.335+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:50.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:50.336+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:11:55.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:11:55.336+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:00.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:00.337+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:05.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:05.338+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:10.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:10.338+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:15.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:15.339+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:20.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:20.340+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:25.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:25.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:30.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:35.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:35.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:40.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:45.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:45.342+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:50.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:50.342+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:12:55.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:12:55.343+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:00.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:00.343+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:05.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:05.344+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:10.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:10.345+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:15.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:15.345+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:20.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:20.346+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:25.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:25.346+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:30.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:30.347+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:35.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:35.348+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:40.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:40.348+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:45.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:45.349+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:50.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:50.350+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:13:55.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:13:55.351+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:00.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:00.351+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:05.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:05.352+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:10.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:10.353+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:15.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:15.352+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:20.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:20.353+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:25.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:25.353+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-09T23:14:30.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:30.354+0000 7f12f4aeb780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:14:30.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:30.354+0000 7f12f4aeb780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:14:30.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:30.354+0000 7f12f4aeb780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:14:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:14:30.375 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... FAIL 2024-04-09T23:14:30.376 DEBUG:boto:path=/xryqqk-60/ 2024-04-09T23:14:30.376 DEBUG:boto:auth_path=/xryqqk-60/ 2024-04-09T23:14:30.377 DEBUG:boto:Method: PUT 2024-04-09T23:14:30.377 DEBUG:boto:Path: /xryqqk-60/ 2024-04-09T23:14:30.377 DEBUG:boto:Data: 2024-04-09T23:14:30.377 DEBUG:boto:Headers: {} 2024-04-09T23:14:30.377 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.377 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.377 DEBUG:boto:Params: {} 2024-04-09T23:14:30.378 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:14:30.378 DEBUG:boto:Token: None 2024-04-09T23:14:30.378 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/ 2024-04-09T23:14:30.378 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lq/dufZIAuvmM0IKNkfvb1XflE4= 2024-04-09T23:14:30.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lq/dufZIAuvmM0IKNkfvb1XflE4=', 'Content-Length': '0'} 2024-04-09T23:14:30.401 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f87ef45d4bc24c8e-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.402 DEBUG:rgw_multi.tests:created bucket=xryqqk-60 2024-04-09T23:14:30.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:14:30.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:14:30.403 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:14:30.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:14:30.457 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:14:30.509 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:14:30.541 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:14:30.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.541 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:14:30.541 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-09T23:14:30.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "0.000000" 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704470.400395_1051.1", 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:14:30.400395Z" 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:14:30.542 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:14:30.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:14:30.543 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:14:30.544 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:14:30.544 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:14:30.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:14:30.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:14:30.544 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:14:30.596 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:14:30.596 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:14:30.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:30.639+0000 7fc2810b6780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:14:30.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:30.640+0000 7fc2810b6780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:14:30.661 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:14:30.661 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.662 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712698963.140128_76.1", 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.663 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "", 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "0.000000", 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704470.400395_1051.1", 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:14:30.400395Z", 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:14:30.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:14:30.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:14:30Z" 2024-04-09T23:14:30.667 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:14:30.668 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712698963.140128_76.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T21:42:43.140128Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712704470.400395_1051.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:14:30.400395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:14:30Z" } 2024-04-09T23:14:30.668 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704470.400395_1051.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:14:30.400395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:14:30.669 DEBUG:rgw_multi.tests:log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:14:30.669 DEBUG:rgw_multi.tests:sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:14:30.669 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:14:30.669 DEBUG:boto:path=/xryqqk-60/ 2024-04-09T23:14:30.669 DEBUG:boto:auth_path=/xryqqk-60/ 2024-04-09T23:14:30.669 DEBUG:boto:Method: HEAD 2024-04-09T23:14:30.669 DEBUG:boto:Path: /xryqqk-60/ 2024-04-09T23:14:30.669 DEBUG:boto:Data: 2024-04-09T23:14:30.669 DEBUG:boto:Headers: {} 2024-04-09T23:14:30.669 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.670 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.670 DEBUG:boto:Params: {} 2024-04-09T23:14:30.670 DEBUG:boto:Token: None 2024-04-09T23:14:30.670 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/ 2024-04-09T23:14:30.670 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-09T23:14:30.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-09T23:14:30.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000168fd2391e264844-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.673 DEBUG:boto:path=/xryqqk-60/obj11 2024-04-09T23:14:30.673 DEBUG:boto:auth_path=/xryqqk-60/obj11 2024-04-09T23:14:30.673 DEBUG:boto:Method: PUT 2024-04-09T23:14:30.673 DEBUG:boto:Path: /xryqqk-60/obj11 2024-04-09T23:14:30.673 DEBUG:boto:Data: 2024-04-09T23:14:30.674 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:14:30.674 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.674 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.674 DEBUG:boto:Params: {} 2024-04-09T23:14:30.674 DEBUG:boto:Token: None 2024-04-09T23:14:30.674 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/obj11 2024-04-09T23:14:30.674 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nC5XBjwtnsIwH7LyWBzHu9r/PVo= 2024-04-09T23:14:30.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nC5XBjwtnsIwH7LyWBzHu9r/PVo='} 2024-04-09T23:14:30.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff2d42046be13aae-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.684 DEBUG:boto:path=/xryqqk-60/ 2024-04-09T23:14:30.684 DEBUG:boto:auth_path=/xryqqk-60/ 2024-04-09T23:14:30.684 DEBUG:boto:Method: HEAD 2024-04-09T23:14:30.684 DEBUG:boto:Path: /xryqqk-60/ 2024-04-09T23:14:30.685 DEBUG:boto:Data: 2024-04-09T23:14:30.685 DEBUG:boto:Headers: {} 2024-04-09T23:14:30.685 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.685 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.685 DEBUG:boto:Params: {} 2024-04-09T23:14:30.685 DEBUG:boto:Token: None 2024-04-09T23:14:30.685 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/ 2024-04-09T23:14:30.685 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-09T23:14:30.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-09T23:14:30.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3021814b0f027d1-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.687 DEBUG:boto:path=/xryqqk-60/obj22 2024-04-09T23:14:30.688 DEBUG:boto:auth_path=/xryqqk-60/obj22 2024-04-09T23:14:30.688 DEBUG:boto:Method: PUT 2024-04-09T23:14:30.688 DEBUG:boto:Path: /xryqqk-60/obj22 2024-04-09T23:14:30.688 DEBUG:boto:Data: 2024-04-09T23:14:30.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:14:30.688 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.688 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.688 DEBUG:boto:Params: {} 2024-04-09T23:14:30.688 DEBUG:boto:Token: None 2024-04-09T23:14:30.688 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/obj22 2024-04-09T23:14:30.689 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9PrSV00GXEJre3CPfNgFAKfwL24= 2024-04-09T23:14:30.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9PrSV00GXEJre3CPfNgFAKfwL24='} 2024-04-09T23:14:30.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026823a0ac558d154-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.697 DEBUG:boto:path=/xryqqk-60/ 2024-04-09T23:14:30.697 DEBUG:boto:auth_path=/xryqqk-60/ 2024-04-09T23:14:30.697 DEBUG:boto:Method: HEAD 2024-04-09T23:14:30.697 DEBUG:boto:Path: /xryqqk-60/ 2024-04-09T23:14:30.698 DEBUG:boto:Data: 2024-04-09T23:14:30.698 DEBUG:boto:Headers: {} 2024-04-09T23:14:30.698 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.698 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.698 DEBUG:boto:Params: {} 2024-04-09T23:14:30.698 DEBUG:boto:Token: None 2024-04-09T23:14:30.698 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/ 2024-04-09T23:14:30.698 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-09T23:14:30.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-09T23:14:30.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000532894accdd0dce7-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.701 DEBUG:boto:path=/xryqqk-60/obj33 2024-04-09T23:14:30.701 DEBUG:boto:auth_path=/xryqqk-60/obj33 2024-04-09T23:14:30.701 DEBUG:boto:Method: PUT 2024-04-09T23:14:30.701 DEBUG:boto:Path: /xryqqk-60/obj33 2024-04-09T23:14:30.701 DEBUG:boto:Data: 2024-04-09T23:14:30.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:14:30.701 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.702 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.702 DEBUG:boto:Params: {} 2024-04-09T23:14:30.702 DEBUG:boto:Token: None 2024-04-09T23:14:30.702 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/obj33 2024-04-09T23:14:30.702 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OIqTdgeskTznUhrKwTqRT951Grc= 2024-04-09T23:14:30.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OIqTdgeskTznUhrKwTqRT951Grc='} 2024-04-09T23:14:30.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009edefdba884e778f-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.747 DEBUG:boto:path=/xryqqk-60/ 2024-04-09T23:14:30.747 DEBUG:boto:auth_path=/xryqqk-60/ 2024-04-09T23:14:30.747 DEBUG:boto:Method: HEAD 2024-04-09T23:14:30.747 DEBUG:boto:Path: /xryqqk-60/ 2024-04-09T23:14:30.747 DEBUG:boto:Data: 2024-04-09T23:14:30.747 DEBUG:boto:Headers: {} 2024-04-09T23:14:30.747 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.747 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.747 DEBUG:boto:Params: {} 2024-04-09T23:14:30.748 DEBUG:boto:Token: None 2024-04-09T23:14:30.748 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/ 2024-04-09T23:14:30.748 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-09T23:14:30.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-09T23:14:30.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000461b41f436a23b09-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.750 DEBUG:boto:path=/xryqqk-60/obj44 2024-04-09T23:14:30.750 DEBUG:boto:auth_path=/xryqqk-60/obj44 2024-04-09T23:14:30.750 DEBUG:boto:Method: PUT 2024-04-09T23:14:30.751 DEBUG:boto:Path: /xryqqk-60/obj44 2024-04-09T23:14:30.751 DEBUG:boto:Data: 2024-04-09T23:14:30.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:14:30.751 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.751 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.751 DEBUG:boto:Params: {} 2024-04-09T23:14:30.751 DEBUG:boto:Token: None 2024-04-09T23:14:30.751 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/obj44 2024-04-09T23:14:30.751 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XVnAZP+DxjGJYAVWSqQUBUS0Ano= 2024-04-09T23:14:30.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XVnAZP+DxjGJYAVWSqQUBUS0Ano='} 2024-04-09T23:14:30.761 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056529d813a7c2b13-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.761 DEBUG:boto:path=/xryqqk-60/ 2024-04-09T23:14:30.761 DEBUG:boto:auth_path=/xryqqk-60/ 2024-04-09T23:14:30.761 DEBUG:boto:Method: HEAD 2024-04-09T23:14:30.761 DEBUG:boto:Path: /xryqqk-60/ 2024-04-09T23:14:30.761 DEBUG:boto:Data: 2024-04-09T23:14:30.762 DEBUG:boto:Headers: {} 2024-04-09T23:14:30.762 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.762 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.762 DEBUG:boto:Params: {} 2024-04-09T23:14:30.762 DEBUG:boto:Token: None 2024-04-09T23:14:30.762 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/ 2024-04-09T23:14:30.762 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-09T23:14:30.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-09T23:14:30.764 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024272e217fef2290-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.765 DEBUG:boto:path=/xryqqk-60/obj55 2024-04-09T23:14:30.765 DEBUG:boto:auth_path=/xryqqk-60/obj55 2024-04-09T23:14:30.765 DEBUG:boto:Method: PUT 2024-04-09T23:14:30.765 DEBUG:boto:Path: /xryqqk-60/obj55 2024-04-09T23:14:30.765 DEBUG:boto:Data: 2024-04-09T23:14:30.765 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:14:30.765 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:14:30.765 DEBUG:boto:Port: 8000 2024-04-09T23:14:30.765 DEBUG:boto:Params: {} 2024-04-09T23:14:30.766 DEBUG:boto:Token: None 2024-04-09T23:14:30.766 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:14:30 GMT /xryqqk-60/obj55 2024-04-09T23:14:30.766 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:x4qnx2TNCfXWs9fDyI3rDJVwfFg= 2024-04-09T23:14:30.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x4qnx2TNCfXWs9fDyI3rDJVwfFg='} 2024-04-09T23:14:30.772 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000430cdf65f80b10db-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:14:30.772 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-60 2024-04-09T23:14:30.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:14:30.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:14:30.772 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:14:30.830 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:14:30.830 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:14:30.879 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-60 2024-04-09T23:14:30.894 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-60 2024-04-09T23:14:30.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:14:30.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:14:30.894 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:14:30.951 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:14:30.951 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:14:30.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:30.988+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:14:35.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:35.989+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:14:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:40.989+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:14:45.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:45.990+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:14:50.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:50.990+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:14:55.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:14:55.991+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:00.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:00.992+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:05.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:05.992+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:10.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:10.993+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:15.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:15.993+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:20.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:20.994+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:25.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:25.995+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:30.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:30.995+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:35.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:35.996+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:40.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:40.997+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:45.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:45.998+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:50.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:50.997+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:15:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:15:55.998+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:00.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:00.998+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:05.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:05.999+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:11.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:10.999+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:16.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:16.000+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:21.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:21.000+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:26.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:26.001+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:31.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:31.001+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:36.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:36.002+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:41.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:41.003+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:46.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:46.003+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:51.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:51.004+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:16:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:16:56.004+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:01.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:01.005+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:06.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:06.006+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:11.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:11.006+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:16.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:16.007+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:21.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:21.008+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:26.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:26.009+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:31.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:31.008+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:36.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:36.009+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:41.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:41.009+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:46.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:46.010+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:51.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:51.011+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:17:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:17:56.011+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:01.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:01.012+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:06.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:06.013+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:11.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:11.013+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:16.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:16.014+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:21.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:21.014+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:26.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:26.015+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:31.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:31.016+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:36.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:36.016+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:41.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:41.017+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:46.017+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:51.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:51.018+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:18:56.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:18:56.018+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:19:01.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:01.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:19:06.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:06.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:19:11.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:11.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:19:16.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:16.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:19:21.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:21.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-09T23:19:26.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.021+0000 7f407fb69780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:19:26.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.021+0000 7f407fb69780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:19:26.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.021+0000 7f407fb69780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:19:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:19:26.043 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... FAIL 2024-04-09T23:19:26.044 INFO:rgw_multi.tests:create role zone=a1 name=roles-xryqqk-1 2024-04-09T23:19:26.044 DEBUG:boto:Method: POST 2024-04-09T23:19:26.044 DEBUG:boto:Path: / 2024-04-09T23:19:26.044 DEBUG:boto:Data: 2024-04-09T23:19:26.044 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.044 DEBUG:boto:Host: smithi063.front.sepia.ceph.com 2024-04-09T23:19:26.044 DEBUG:boto:Port: 8000 2024-04-09T23:19:26.044 DEBUG:boto:Params: {'RoleName': 'roles-xryqqk-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-09T23:19:26.045 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:19:26.045 DEBUG:boto:Token: None 2024-04-09T23:19:26.045 DEBUG:boto:CanonicalRequest: POST / host:smithi063.front.sepia.ceph.com:8000 x-amz-date:20240409T231926Z host;x-amz-date 0386d9b4564e0130cbb38ada582bebc2e4208ac43288c8b8e23c2206ac4749d3 2024-04-09T23:19:26.045 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240409T231926Z 20240409/front/smithi063/aws4_request adf0375a2b32e13c73b7c2135354937acc968b66d62d218b81abc1c2e1efa219 2024-04-09T23:19:26.046 DEBUG:boto:Signature: 577f578efae8e591f3911776daf7de1b7204122ea7101b728ed3ef12b850b519 2024-04-09T23:19:26.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240409T231926Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=N5OV6KWZ8WWR3JMWMG4W/20240409/front/smithi063/aws4_request,SignedHeaders=host;x-amz-date,Signature=577f578efae8e591f3911776daf7de1b7204122ea7101b728ed3ef12b850b519', 'Host': 'smithi063.front.sepia.ceph.com:8000'} 2024-04-09T23:19:26.061 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000856a0702d23ea29c-006615ccfe-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '722'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.061 DEBUG:boto:b'932aaba0-bb34-4f69-a68d-654af092a1e9roles-xryqqk-1/arn:aws:iam:::role/roles-xryqqk-12024-04-09T23:19:26.52Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000856a0702d23ea29c-006615ccfe-1b6d-a1' 2024-04-09T23:19:26.062 INFO:rgw_multi.tests:create role zone=a2 name=roles-xryqqk-2 2024-04-09T23:19:26.062 DEBUG:boto:Method: POST 2024-04-09T23:19:26.062 DEBUG:boto:Path: / 2024-04-09T23:19:26.062 DEBUG:boto:Data: 2024-04-09T23:19:26.062 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.063 DEBUG:boto:Host: smithi063.front.sepia.ceph.com 2024-04-09T23:19:26.063 DEBUG:boto:Port: 8001 2024-04-09T23:19:26.063 DEBUG:boto:Params: {'RoleName': 'roles-xryqqk-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-09T23:19:26.063 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:19:26.063 DEBUG:boto:Token: None 2024-04-09T23:19:26.063 DEBUG:boto:CanonicalRequest: POST / host:smithi063.front.sepia.ceph.com:8001 x-amz-date:20240409T231926Z host;x-amz-date 374d66740c85f50250ce809326b8c82002132c69b1f4926031607ab2b585d27d 2024-04-09T23:19:26.063 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240409T231926Z 20240409/front/smithi063/aws4_request 5400847955505c5d8483f2061a3a39a6ba92cc7b7e95926ee1b6def38a11b0df 2024-04-09T23:19:26.063 DEBUG:boto:Signature: a70e3a68dc842c97bb856f6e41196b3bfd058cb2df1246b6a47b6d0dc8dfd64a 2024-04-09T23:19:26.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240409T231926Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=N5OV6KWZ8WWR3JMWMG4W/20240409/front/smithi063/aws4_request,SignedHeaders=host;x-amz-date,Signature=a70e3a68dc842c97bb856f6e41196b3bfd058cb2df1246b6a47b6d0dc8dfd64a', 'Host': 'smithi063.front.sepia.ceph.com:8001'} 2024-04-09T23:19:26.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e55428285aeb40b-006615ccfe-1fc6-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '722'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.092 DEBUG:boto:b'11d5a4d9-8f56-4f84-9436-a80c3f0dca7broles-xryqqk-2/arn:aws:iam:::role/roles-xryqqk-22024-04-09T23:19:26.85Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000002e55428285aeb40b-006615ccfe-1fc6-a2' 2024-04-09T23:19:26.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.093 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:19:26.161 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:26.161 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:26.207 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:19:26.222 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:19:26.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.222 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704470.400395_1051.1", 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:14:30.400395Z" 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:19:26.223 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:19:26.224 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.224 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:19:26.225 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:19:26.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:19:26.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.225 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:19:26.275 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:26.275 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:26.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.310+0000 7fcca0162780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:19:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.311+0000 7fcca0162780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:19:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.326 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.327 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704470.400395_1051.1", 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:14:30.400395Z", 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.329 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:19:26.329 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.329 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:19:26.330 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:19:26.331 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:19:26.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.331 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:19:26Z" 2024-04-09T23:19:26.331 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:19:26.332 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712704470.400395_1051.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:14:30.400395Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:19:26Z" } 2024-04-09T23:19:26.332 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704470.400395_1051.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:14:30.400395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:19:26.332 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:19:26.332 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:19:26.332 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:19:26.332 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xryqqk-1 zones={a2, a1} 2024-04-09T23:19:26.332 DEBUG:boto:Method: POST 2024-04-09T23:19:26.332 DEBUG:boto:Path: / 2024-04-09T23:19:26.333 DEBUG:boto:Data: 2024-04-09T23:19:26.333 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.333 DEBUG:boto:Host: smithi063.front.sepia.ceph.com 2024-04-09T23:19:26.333 DEBUG:boto:Port: 8001 2024-04-09T23:19:26.333 DEBUG:boto:Params: {'RoleName': 'roles-xryqqk-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-09T23:19:26.333 DEBUG:boto:Token: None 2024-04-09T23:19:26.333 DEBUG:boto:CanonicalRequest: POST / host:smithi063.front.sepia.ceph.com:8001 x-amz-date:20240409T231926Z host;x-amz-date aa2e7ca9f2b39673871687ddf6ad66ca6c8641b98f27b41ea384a32713fccb99 2024-04-09T23:19:26.333 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240409T231926Z 20240409/front/smithi063/aws4_request 45385aa9eb456222d64612fca8b915bd90a7f133a5facc64911699eb6b53b5a5 2024-04-09T23:19:26.333 DEBUG:boto:Signature: 96b4f1bf7be2e41d671e91e98965c582a0b079e3ad87ed7fd8c0a345ff8384b6 2024-04-09T23:19:26.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240409T231926Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=N5OV6KWZ8WWR3JMWMG4W/20240409/front/smithi063/aws4_request,SignedHeaders=host;x-amz-date,Signature=96b4f1bf7be2e41d671e91e98965c582a0b079e3ad87ed7fd8c0a345ff8384b6', 'Host': 'smithi063.front.sepia.ceph.com:8001'} 2024-04-09T23:19:26.337 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000488bfe411b9412cc-006615ccfe-1fc6-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.337 DEBUG:boto:b'tx00000488bfe411b9412cc-006615ccfe-1fc6-a2932aaba0-bb34-4f69-a68d-654af092a1e9roles-xryqqk-1/arn:aws:iam:::role/roles-xryqqk-12024-04-09T23:19:26.123Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-09T23:19:26.338 DEBUG:boto:Method: POST 2024-04-09T23:19:26.338 DEBUG:boto:Path: / 2024-04-09T23:19:26.338 DEBUG:boto:Data: 2024-04-09T23:19:26.338 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.339 DEBUG:boto:Host: smithi063.front.sepia.ceph.com 2024-04-09T23:19:26.339 DEBUG:boto:Port: 8000 2024-04-09T23:19:26.339 DEBUG:boto:Params: {'RoleName': 'roles-xryqqk-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-09T23:19:26.339 DEBUG:boto:Token: None 2024-04-09T23:19:26.339 DEBUG:boto:CanonicalRequest: POST / host:smithi063.front.sepia.ceph.com:8000 x-amz-date:20240409T231926Z host;x-amz-date aa2e7ca9f2b39673871687ddf6ad66ca6c8641b98f27b41ea384a32713fccb99 2024-04-09T23:19:26.339 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240409T231926Z 20240409/front/smithi063/aws4_request fe98b41e1c259d2d3b72fdae9e5a5f888574e87aa54576632d08412876321bfb 2024-04-09T23:19:26.339 DEBUG:boto:Signature: ab60032c98cf778f2a565227dab3a0478da4d0aff895dc351e3fb0236e203a33 2024-04-09T23:19:26.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240409T231926Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=N5OV6KWZ8WWR3JMWMG4W/20240409/front/smithi063/aws4_request,SignedHeaders=host;x-amz-date,Signature=ab60032c98cf778f2a565227dab3a0478da4d0aff895dc351e3fb0236e203a33', 'Host': 'smithi063.front.sepia.ceph.com:8000'} 2024-04-09T23:19:26.341 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b66e39e823813d60-006615ccfe-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '710'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.341 DEBUG:boto:b'tx00000b66e39e823813d60-006615ccfe-1b6d-a1932aaba0-bb34-4f69-a68d-654af092a1e9roles-xryqqk-1/arn:aws:iam:::role/roles-xryqqk-12024-04-09T23:19:26.52Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-09T23:19:26.341 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xryqqk-1 2024-04-09T23:19:26.341 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xryqqk-1 zones={a2, a1} 2024-04-09T23:19:26.341 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-xryqqk-2 zones={a1, a2} 2024-04-09T23:19:26.342 DEBUG:boto:Method: POST 2024-04-09T23:19:26.342 DEBUG:boto:Path: / 2024-04-09T23:19:26.342 DEBUG:boto:Data: 2024-04-09T23:19:26.342 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.342 DEBUG:boto:Host: smithi063.front.sepia.ceph.com 2024-04-09T23:19:26.342 DEBUG:boto:Port: 8000 2024-04-09T23:19:26.342 DEBUG:boto:Params: {'RoleName': 'roles-xryqqk-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-09T23:19:26.342 DEBUG:boto:Token: None 2024-04-09T23:19:26.342 DEBUG:boto:CanonicalRequest: POST / host:smithi063.front.sepia.ceph.com:8000 x-amz-date:20240409T231926Z host;x-amz-date 2c92bf30afb3bb6d099c6eb3feb7bcf56bc3f084e53c437a6d1fc13ab80f624f 2024-04-09T23:19:26.343 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240409T231926Z 20240409/front/smithi063/aws4_request ff29d2a9082d91c1a6d8597fb1df9f970eb0159cdfb20061601806b017a6277c 2024-04-09T23:19:26.343 DEBUG:boto:Signature: d17ab67f46645a829e458bc1c88766b57c988d5f67546a491d88dd2d4ddd8cb1 2024-04-09T23:19:26.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240409T231926Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=N5OV6KWZ8WWR3JMWMG4W/20240409/front/smithi063/aws4_request,SignedHeaders=host;x-amz-date,Signature=d17ab67f46645a829e458bc1c88766b57c988d5f67546a491d88dd2d4ddd8cb1', 'Host': 'smithi063.front.sepia.ceph.com:8000'} 2024-04-09T23:19:26.344 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b7d9a34503784ab7-006615ccfe-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '710'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.344 DEBUG:boto:b'tx00000b7d9a34503784ab7-006615ccfe-1b6d-a111d5a4d9-8f56-4f84-9436-a80c3f0dca7broles-xryqqk-2/arn:aws:iam:::role/roles-xryqqk-22024-04-09T23:19:26.71Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-09T23:19:26.344 DEBUG:boto:Method: POST 2024-04-09T23:19:26.345 DEBUG:boto:Path: / 2024-04-09T23:19:26.345 DEBUG:boto:Data: 2024-04-09T23:19:26.345 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.345 DEBUG:boto:Host: smithi063.front.sepia.ceph.com 2024-04-09T23:19:26.345 DEBUG:boto:Port: 8001 2024-04-09T23:19:26.345 DEBUG:boto:Params: {'RoleName': 'roles-xryqqk-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-09T23:19:26.345 DEBUG:boto:Token: None 2024-04-09T23:19:26.345 DEBUG:boto:CanonicalRequest: POST / host:smithi063.front.sepia.ceph.com:8001 x-amz-date:20240409T231926Z host;x-amz-date 2c92bf30afb3bb6d099c6eb3feb7bcf56bc3f084e53c437a6d1fc13ab80f624f 2024-04-09T23:19:26.345 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240409T231926Z 20240409/front/smithi063/aws4_request 12479779fd74850ac6e6ed6d457a409b43b03487b1dc0bcc8e88f5b06a6ba73a 2024-04-09T23:19:26.346 DEBUG:boto:Signature: 3dbe9393ad283a0ec9b056ce4c1ae6614b7daa90bd9a80a71aa8f194c30024aa 2024-04-09T23:19:26.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240409T231926Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=N5OV6KWZ8WWR3JMWMG4W/20240409/front/smithi063/aws4_request,SignedHeaders=host;x-amz-date,Signature=3dbe9393ad283a0ec9b056ce4c1ae6614b7daa90bd9a80a71aa8f194c30024aa', 'Host': 'smithi063.front.sepia.ceph.com:8001'} 2024-04-09T23:19:26.347 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000046cf56269105e9db-006615ccfe-1fc6-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '710'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.347 DEBUG:boto:b'tx0000046cf56269105e9db-006615ccfe-1fc6-a211d5a4d9-8f56-4f84-9436-a80c3f0dca7broles-xryqqk-2/arn:aws:iam:::role/roles-xryqqk-22024-04-09T23:19:26.71Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-09T23:19:26.348 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-xryqqk-2 2024-04-09T23:19:26.348 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-xryqqk-2 zones={a1, a2} 2024-04-09T23:19:26.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-04-09T23:19:26.349 DEBUG:boto:path=/xryqqk-61/ 2024-04-09T23:19:26.349 DEBUG:boto:auth_path=/xryqqk-61/ 2024-04-09T23:19:26.349 DEBUG:boto:Method: PUT 2024-04-09T23:19:26.349 DEBUG:boto:Path: /xryqqk-61/ 2024-04-09T23:19:26.349 DEBUG:boto:Data: 2024-04-09T23:19:26.349 DEBUG:boto:Headers: {} 2024-04-09T23:19:26.349 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:26.350 DEBUG:boto:Port: 8000 2024-04-09T23:19:26.350 DEBUG:boto:Params: {} 2024-04-09T23:19:26.350 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:19:26.350 DEBUG:boto:Token: None 2024-04-09T23:19:26.350 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:19:26 GMT /xryqqk-61/ 2024-04-09T23:19:26.350 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3urQGhh6NcnIzSRU3gLwjFHC8x4= 2024-04-09T23:19:26.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:26 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3urQGhh6NcnIzSRU3gLwjFHC8x4=', 'Content-Length': '0'} 2024-04-09T23:19:26.369 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f595e70eb2ada1f-006615ccfe-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:26.369 DEBUG:rgw_multi.tests:created bucket=xryqqk-61 2024-04-09T23:19:26.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.369 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:19:26.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:26.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:26.452 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:19:26.467 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:19:26.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.368163_1115.1", 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.368163Z" 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.469 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:19:26.469 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:19:26.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.469 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:19:26.470 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704766.368163_1115.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:19:26.470 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:19:26.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:19:26.470 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:19:26.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:26.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:26.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.578+0000 7f3000cbe780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:19:26.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:26.579+0000 7f3000cbe780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:19:26.601 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:19:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:19:26.601 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.602 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:19:26.603 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:19:26.604 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.368163_1115.1", 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.368163Z", 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:19:26.605 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:19:26.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:19:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:19:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:19:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:19:26.607 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:19:26Z" 2024-04-09T23:19:26.607 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:19:26.608 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712704766.368163_1115.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.368163Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:19:26Z" } 2024-04-09T23:19:26.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704766.368163_1115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.368163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:19:26.608 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704766.368163_1115.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:19:26.608 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704766.368163_1115.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:19:26.608 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:19:26.609 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-09T23:19:26.609 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T23:19:26.609 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:19:26.609+0000 7f4866209700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 101072) UID: 0 2024-04-09T23:19:26.610 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:19:26.609+0000 7f486b3de7c0 -1 shutting down 2024-04-09T23:19:31.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-09T23:19:32.711 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-09T23:19:32.711 DEBUG:boto:path=/xryqqk-61/ 2024-04-09T23:19:32.711 DEBUG:boto:auth_path=/xryqqk-61/ 2024-04-09T23:19:32.712 DEBUG:boto:Method: HEAD 2024-04-09T23:19:32.712 DEBUG:boto:Path: /xryqqk-61/ 2024-04-09T23:19:32.712 DEBUG:boto:Data: 2024-04-09T23:19:32.712 DEBUG:boto:Headers: {} 2024-04-09T23:19:32.712 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.713 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.713 DEBUG:boto:Params: {} 2024-04-09T23:19:32.713 DEBUG:boto:Token: None 2024-04-09T23:19:32.714 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/ 2024-04-09T23:19:32.714 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-09T23:19:32.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-09T23:19:32.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000596a7ab655943e91-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.718 DEBUG:boto:path=/xryqqk-61/obj11 2024-04-09T23:19:32.718 DEBUG:boto:auth_path=/xryqqk-61/obj11 2024-04-09T23:19:32.719 DEBUG:boto:Method: PUT 2024-04-09T23:19:32.719 DEBUG:boto:Path: /xryqqk-61/obj11 2024-04-09T23:19:32.719 DEBUG:boto:Data: 2024-04-09T23:19:32.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:19:32.720 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.720 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.720 DEBUG:boto:Params: {} 2024-04-09T23:19:32.721 DEBUG:boto:Token: None 2024-04-09T23:19:32.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/obj11 2024-04-09T23:19:32.722 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TqPEeIMRTGpqIKzWrd/6G9DP4Mc= 2024-04-09T23:19:32.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TqPEeIMRTGpqIKzWrd/6G9DP4Mc='} 2024-04-09T23:19:32.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054f1333d7af2a600-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.730 DEBUG:boto:path=/xryqqk-61/ 2024-04-09T23:19:32.731 DEBUG:boto:auth_path=/xryqqk-61/ 2024-04-09T23:19:32.731 DEBUG:boto:Method: HEAD 2024-04-09T23:19:32.731 DEBUG:boto:Path: /xryqqk-61/ 2024-04-09T23:19:32.731 DEBUG:boto:Data: 2024-04-09T23:19:32.731 DEBUG:boto:Headers: {} 2024-04-09T23:19:32.732 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.732 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.732 DEBUG:boto:Params: {} 2024-04-09T23:19:32.732 DEBUG:boto:Token: None 2024-04-09T23:19:32.733 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/ 2024-04-09T23:19:32.733 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-09T23:19:32.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-09T23:19:32.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e3f6609af8d0ac1-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.736 DEBUG:boto:path=/xryqqk-61/obj22 2024-04-09T23:19:32.736 DEBUG:boto:auth_path=/xryqqk-61/obj22 2024-04-09T23:19:32.737 DEBUG:boto:Method: PUT 2024-04-09T23:19:32.737 DEBUG:boto:Path: /xryqqk-61/obj22 2024-04-09T23:19:32.737 DEBUG:boto:Data: 2024-04-09T23:19:32.737 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:19:32.738 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.738 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.738 DEBUG:boto:Params: {} 2024-04-09T23:19:32.738 DEBUG:boto:Token: None 2024-04-09T23:19:32.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/obj22 2024-04-09T23:19:32.739 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2bftUdGYQ9hgTQc51O92gD+mZBc= 2024-04-09T23:19:32.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2bftUdGYQ9hgTQc51O92gD+mZBc='} 2024-04-09T23:19:32.744 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f419afeab475e773-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.744 DEBUG:boto:path=/xryqqk-61/ 2024-04-09T23:19:32.745 DEBUG:boto:auth_path=/xryqqk-61/ 2024-04-09T23:19:32.745 DEBUG:boto:Method: HEAD 2024-04-09T23:19:32.745 DEBUG:boto:Path: /xryqqk-61/ 2024-04-09T23:19:32.745 DEBUG:boto:Data: 2024-04-09T23:19:32.746 DEBUG:boto:Headers: {} 2024-04-09T23:19:32.746 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.746 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.746 DEBUG:boto:Params: {} 2024-04-09T23:19:32.747 DEBUG:boto:Token: None 2024-04-09T23:19:32.747 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/ 2024-04-09T23:19:32.747 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-09T23:19:32.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-09T23:19:32.750 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092be558bd0cdc088-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.751 DEBUG:boto:path=/xryqqk-61/obj33 2024-04-09T23:19:32.751 DEBUG:boto:auth_path=/xryqqk-61/obj33 2024-04-09T23:19:32.751 DEBUG:boto:Method: PUT 2024-04-09T23:19:32.751 DEBUG:boto:Path: /xryqqk-61/obj33 2024-04-09T23:19:32.751 DEBUG:boto:Data: 2024-04-09T23:19:32.751 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:19:32.751 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.751 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.751 DEBUG:boto:Params: {} 2024-04-09T23:19:32.752 DEBUG:boto:Token: None 2024-04-09T23:19:32.752 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/obj33 2024-04-09T23:19:32.752 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5d1XpjqbXVS5nwlI6BLF2ibObwo= 2024-04-09T23:19:32.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5d1XpjqbXVS5nwlI6BLF2ibObwo='} 2024-04-09T23:19:32.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b2298938c13f29e-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.795 DEBUG:boto:path=/xryqqk-61/ 2024-04-09T23:19:32.795 DEBUG:boto:auth_path=/xryqqk-61/ 2024-04-09T23:19:32.795 DEBUG:boto:Method: HEAD 2024-04-09T23:19:32.795 DEBUG:boto:Path: /xryqqk-61/ 2024-04-09T23:19:32.795 DEBUG:boto:Data: 2024-04-09T23:19:32.795 DEBUG:boto:Headers: {} 2024-04-09T23:19:32.796 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.796 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.796 DEBUG:boto:Params: {} 2024-04-09T23:19:32.796 DEBUG:boto:Token: None 2024-04-09T23:19:32.796 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/ 2024-04-09T23:19:32.796 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-09T23:19:32.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-09T23:19:32.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005acccc590bc890f-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.799 DEBUG:boto:path=/xryqqk-61/obj44 2024-04-09T23:19:32.799 DEBUG:boto:auth_path=/xryqqk-61/obj44 2024-04-09T23:19:32.800 DEBUG:boto:Method: PUT 2024-04-09T23:19:32.800 DEBUG:boto:Path: /xryqqk-61/obj44 2024-04-09T23:19:32.800 DEBUG:boto:Data: 2024-04-09T23:19:32.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:19:32.800 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.800 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.800 DEBUG:boto:Params: {} 2024-04-09T23:19:32.800 DEBUG:boto:Token: None 2024-04-09T23:19:32.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/obj44 2024-04-09T23:19:32.801 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:O965TUr4uWgje6omCB7u9isY1iE= 2024-04-09T23:19:32.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:O965TUr4uWgje6omCB7u9isY1iE='} 2024-04-09T23:19:32.811 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ae3259341e51d6e-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.812 DEBUG:boto:path=/xryqqk-61/ 2024-04-09T23:19:32.812 DEBUG:boto:auth_path=/xryqqk-61/ 2024-04-09T23:19:32.812 DEBUG:boto:Method: HEAD 2024-04-09T23:19:32.812 DEBUG:boto:Path: /xryqqk-61/ 2024-04-09T23:19:32.812 DEBUG:boto:Data: 2024-04-09T23:19:32.812 DEBUG:boto:Headers: {} 2024-04-09T23:19:32.812 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.812 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.813 DEBUG:boto:Params: {} 2024-04-09T23:19:32.813 DEBUG:boto:Token: None 2024-04-09T23:19:32.813 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/ 2024-04-09T23:19:32.813 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-09T23:19:32.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-09T23:19:32.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0dc3796901d7bba-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.816 DEBUG:boto:path=/xryqqk-61/obj55 2024-04-09T23:19:32.816 DEBUG:boto:auth_path=/xryqqk-61/obj55 2024-04-09T23:19:32.817 DEBUG:boto:Method: PUT 2024-04-09T23:19:32.817 DEBUG:boto:Path: /xryqqk-61/obj55 2024-04-09T23:19:32.817 DEBUG:boto:Data: 2024-04-09T23:19:32.817 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:19:32.817 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:19:32.817 DEBUG:boto:Port: 8000 2024-04-09T23:19:32.817 DEBUG:boto:Params: {} 2024-04-09T23:19:32.817 DEBUG:boto:Token: None 2024-04-09T23:19:32.818 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:19:32 GMT /xryqqk-61/obj55 2024-04-09T23:19:32.818 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gMdB0DMgROcSkMT7H/htiZ/xCWU= 2024-04-09T23:19:32.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gMdB0DMgROcSkMT7H/htiZ/xCWU='} 2024-04-09T23:19:32.867 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa81b19ac7637ae5-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:19:32.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:19:32.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:19:32.868 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:19:32.926 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:32.927 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:32.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:32.967+0000 7fdb09d6f780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-09T23:19:32.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:32.969+0000 7fdb09d6f780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-09T23:19:32.994 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-09T23:19:32.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-09T23:19:32.996 INFO:tasks.rgw.c1.client.1:Started 2024-04-09T23:19:32.996 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T23:19:33.018 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T23:19:33.018 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T23:19:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T23:19:33.020 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-09T23:19:34.021 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T23:19:34.043 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T23:19:34.043 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T23:19:34.058 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 14266 0 --:--:-- --:--:-- --:--:-- 14266 2024-04-09T23:19:34.059 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T23:19:34.059 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-61 2024-04-09T23:19:34.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:19:34.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:19:34.060 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:19:34.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:34.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:34.194 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-61 2024-04-09T23:19:34.214 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-61 2024-04-09T23:19:34.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:19:34.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:19:34.215 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:19:34.272 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:19:34.273 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:19:34.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:34.329+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:19:37.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-09T23:19:39.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:39.330+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:19:44.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:44.330+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:19:49.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:49.331+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:19:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:54.331+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:19:59.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:19:59.331+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:04.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:04.332+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:09.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:09.333+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:14.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:14.333+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:19.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:19.334+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:24.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:24.334+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:29.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:29.335+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:34.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:34.336+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:39.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:39.337+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:44.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:44.338+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:49.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:49.337+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:54.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:54.338+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:20:59.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:20:59.339+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:04.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:04.339+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:09.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:09.340+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:14.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:14.340+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:19.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:19.341+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:24.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:24.341+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:29.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:29.342+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:34.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:34.342+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:39.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:39.343+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:44.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:44.343+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:49.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:49.344+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:54.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:54.345+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:21:59.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:21:59.345+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:04.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:04.346+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:09.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:09.346+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:14.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:14.347+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:19.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:19.348+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:24.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:24.348+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:29.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:29.348+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:34.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:34.349+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:39.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:39.349+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:44.350+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:49.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:49.350+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:54.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:54.351+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:22:59.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:22:59.352+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:04.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:04.352+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:09.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:09.353+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:14.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:14.353+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:19.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:19.354+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:24.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:24.355+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:29.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:29.355+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:34.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:34.356+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:39.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:39.356+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:44.357+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:49.357+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:54.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:54.358+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:23:59.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:23:59.359+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:24:04.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:04.359+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:24:09.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:09.360+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:24:14.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:14.360+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:24:19.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:19.361+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:24:24.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:24.361+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-09T23:24:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:29.362+0000 7f3d18315780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:24:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:29.362+0000 7f3d18315780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:24:29.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:29.362+0000 7f3d18315780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:24:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:24:29.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... FAIL 2024-04-09T23:24:29.385 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:29.385 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:29.385 DEBUG:boto:Method: PUT 2024-04-09T23:24:29.385 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:29.385 DEBUG:boto:Data: 2024-04-09T23:24:29.385 DEBUG:boto:Headers: {} 2024-04-09T23:24:29.385 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:29.386 DEBUG:boto:Port: 8000 2024-04-09T23:24:29.386 DEBUG:boto:Params: {} 2024-04-09T23:24:29.386 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:24:29.386 DEBUG:boto:Token: None 2024-04-09T23:24:29.387 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:24:29 GMT /xryqqk-62/ 2024-04-09T23:24:29.387 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uz3ibZJRqzxQj7qgrh0vzMBjsxA= 2024-04-09T23:24:29.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:29 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uz3ibZJRqzxQj7qgrh0vzMBjsxA=', 'Content-Length': '0'} 2024-04-09T23:24:29.411 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000954e640745db255c-006615ce2d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:29.411 DEBUG:rgw_multi.tests:created bucket=xryqqk-62 2024-04-09T23:24:29.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:24:29.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:24:29.412 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:24:29.485 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:29.485 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:29.527 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:24:29.545 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:24:29.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.545 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705069.409525_1169.1", 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:24:29.409525Z" 2024-04-09T23:24:29.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.547 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:24:29.547 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:24:29.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:29.547 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:24:29.548 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:24:29.548 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:24:29.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:24:29.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:24:29.549 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:24:29.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:29.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:29.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:29.645+0000 7fc454565780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:24:29.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:29.646+0000 7fc454565780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:24:29.666 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:24:29.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:29.668 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:24:29.669 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.670 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705069.400619_1166.1", 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:24:29.400619Z", 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:29.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:24:29.672 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:24:29.673 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:24:29.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:29.674 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:24:29Z" 2024-04-09T23:24:29.674 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:24:29.675 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705069.400619_1166.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:24:29.400619Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:24:29Z" } 2024-04-09T23:24:29.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705069.400619_1166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:24:29.400619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:24:29.675 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:24:29.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.400619_1166.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:24:29.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712705069.409525_1169.1 target=1_1712705069.400619_1166.1 2024-04-09T23:24:34.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:24:34.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:24:34.677 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:24:34.736 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:34.737 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:34.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:34.793+0000 7f3f835f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:24:34.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:34.795+0000 7f3f835f5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:24:34.811 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:24:34.811 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:24:34.811 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:24:34.812 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:24:34.812 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:24:34.812 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:24:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:34.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:34.815 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705069.409525_1169.1", 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:24:29.409525Z", 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:24:34.816 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:34.817 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:24:34.818 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:24:34.818 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:24:34.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:24:34.818 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:24:34Z" 2024-04-09T23:24:34.818 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:24:34.819 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705069.409525_1169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:24:29.409525Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:24:34Z" } 2024-04-09T23:24:34.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705069.409525_1169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:24:29.409525Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:24:34.819 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:24:34.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:24:34.819 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:24:34.820 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-09T23:24:34.820 INFO:teuthology.orchestra.run:waiting for 300 2024-04-09T23:24:34.821 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:24:34.820+0000 7f2561ab8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 103748) UID: 0 2024-04-09T23:24:34.821 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:24:34.820+0000 7f2566c8d7c0 -1 shutting down 2024-04-09T23:24:39.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-09T23:24:40.921 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-09T23:24:40.921 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:40.922 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:40.922 DEBUG:boto:Method: HEAD 2024-04-09T23:24:40.922 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:40.922 DEBUG:boto:Data: 2024-04-09T23:24:40.922 DEBUG:boto:Headers: {} 2024-04-09T23:24:40.922 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.922 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.922 DEBUG:boto:Params: {} 2024-04-09T23:24:40.922 DEBUG:boto:Token: None 2024-04-09T23:24:40.923 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/ 2024-04-09T23:24:40.923 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-09T23:24:40.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-09T23:24:40.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cd14c5ace93f8b2-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.926 DEBUG:boto:path=/xryqqk-62/obj11 2024-04-09T23:24:40.926 DEBUG:boto:auth_path=/xryqqk-62/obj11 2024-04-09T23:24:40.926 DEBUG:boto:Method: PUT 2024-04-09T23:24:40.926 DEBUG:boto:Path: /xryqqk-62/obj11 2024-04-09T23:24:40.926 DEBUG:boto:Data: 2024-04-09T23:24:40.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:40.926 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.926 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.926 DEBUG:boto:Params: {} 2024-04-09T23:24:40.927 DEBUG:boto:Token: None 2024-04-09T23:24:40.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/obj11 2024-04-09T23:24:40.927 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:s2R5E5C//4lsLIXQ68IY20gMl3g= 2024-04-09T23:24:40.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:s2R5E5C//4lsLIXQ68IY20gMl3g='} 2024-04-09T23:24:40.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082d8f42157a5c64b-006615ce38-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.936 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:40.936 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:40.936 DEBUG:boto:Method: HEAD 2024-04-09T23:24:40.936 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:40.937 DEBUG:boto:Data: 2024-04-09T23:24:40.937 DEBUG:boto:Headers: {} 2024-04-09T23:24:40.937 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.937 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.937 DEBUG:boto:Params: {} 2024-04-09T23:24:40.937 DEBUG:boto:Token: None 2024-04-09T23:24:40.937 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/ 2024-04-09T23:24:40.937 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-09T23:24:40.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-09T23:24:40.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7e90f816c51905c-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.940 DEBUG:boto:path=/xryqqk-62/obj22 2024-04-09T23:24:40.940 DEBUG:boto:auth_path=/xryqqk-62/obj22 2024-04-09T23:24:40.940 DEBUG:boto:Method: PUT 2024-04-09T23:24:40.940 DEBUG:boto:Path: /xryqqk-62/obj22 2024-04-09T23:24:40.940 DEBUG:boto:Data: 2024-04-09T23:24:40.940 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:40.940 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.940 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.940 DEBUG:boto:Params: {} 2024-04-09T23:24:40.941 DEBUG:boto:Token: None 2024-04-09T23:24:40.941 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/obj22 2024-04-09T23:24:40.941 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:O4GiUvFc8ezkWe8+11qbt+y8Xbo= 2024-04-09T23:24:40.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:O4GiUvFc8ezkWe8+11qbt+y8Xbo='} 2024-04-09T23:24:40.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7962aaac9415477-006615ce38-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.945 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:40.945 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:40.946 DEBUG:boto:Method: HEAD 2024-04-09T23:24:40.946 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:40.946 DEBUG:boto:Data: 2024-04-09T23:24:40.946 DEBUG:boto:Headers: {} 2024-04-09T23:24:40.946 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.946 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.946 DEBUG:boto:Params: {} 2024-04-09T23:24:40.946 DEBUG:boto:Token: None 2024-04-09T23:24:40.946 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/ 2024-04-09T23:24:40.946 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-09T23:24:40.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-09T23:24:40.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83212c03e156bce-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.949 DEBUG:boto:path=/xryqqk-62/obj33 2024-04-09T23:24:40.949 DEBUG:boto:auth_path=/xryqqk-62/obj33 2024-04-09T23:24:40.949 DEBUG:boto:Method: PUT 2024-04-09T23:24:40.949 DEBUG:boto:Path: /xryqqk-62/obj33 2024-04-09T23:24:40.949 DEBUG:boto:Data: 2024-04-09T23:24:40.950 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:40.950 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.950 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.950 DEBUG:boto:Params: {} 2024-04-09T23:24:40.950 DEBUG:boto:Token: None 2024-04-09T23:24:40.950 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/obj33 2024-04-09T23:24:40.950 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uD3/bDzRANsk2RHW+fP52Fltoqc= 2024-04-09T23:24:40.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uD3/bDzRANsk2RHW+fP52Fltoqc='} 2024-04-09T23:24:40.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f1f9fd74e268be-006615ce38-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.995 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:40.995 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:40.995 DEBUG:boto:Method: HEAD 2024-04-09T23:24:40.995 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:40.995 DEBUG:boto:Data: 2024-04-09T23:24:40.995 DEBUG:boto:Headers: {} 2024-04-09T23:24:40.995 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.995 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.995 DEBUG:boto:Params: {} 2024-04-09T23:24:40.996 DEBUG:boto:Token: None 2024-04-09T23:24:40.996 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/ 2024-04-09T23:24:40.996 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-09T23:24:40.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-09T23:24:40.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068f761486e4bec5e-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:40.998 DEBUG:boto:path=/xryqqk-62/obj44 2024-04-09T23:24:40.998 DEBUG:boto:auth_path=/xryqqk-62/obj44 2024-04-09T23:24:40.998 DEBUG:boto:Method: PUT 2024-04-09T23:24:40.998 DEBUG:boto:Path: /xryqqk-62/obj44 2024-04-09T23:24:40.998 DEBUG:boto:Data: 2024-04-09T23:24:40.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:40.999 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:40.999 DEBUG:boto:Port: 8000 2024-04-09T23:24:40.999 DEBUG:boto:Params: {} 2024-04-09T23:24:40.999 DEBUG:boto:Token: None 2024-04-09T23:24:40.999 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:40 GMT /xryqqk-62/obj44 2024-04-09T23:24:40.999 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hXOkIVCfE8wtwE45MRJROQlEFE0= 2024-04-09T23:24:41.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hXOkIVCfE8wtwE45MRJROQlEFE0='} 2024-04-09T23:24:41.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cde588f179708dfe-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.010 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.011 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.011 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.011 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.011 DEBUG:boto:Data: 2024-04-09T23:24:41.011 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.011 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.011 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.011 DEBUG:boto:Params: {} 2024-04-09T23:24:41.012 DEBUG:boto:Token: None 2024-04-09T23:24:41.012 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.012 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fcb5ef84262ff98-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.014 DEBUG:boto:path=/xryqqk-62/obj55 2024-04-09T23:24:41.014 DEBUG:boto:auth_path=/xryqqk-62/obj55 2024-04-09T23:24:41.015 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.015 DEBUG:boto:Path: /xryqqk-62/obj55 2024-04-09T23:24:41.015 DEBUG:boto:Data: 2024-04-09T23:24:41.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.015 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.015 DEBUG:boto:Params: {} 2024-04-09T23:24:41.015 DEBUG:boto:Token: None 2024-04-09T23:24:41.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj55 2024-04-09T23:24:41.016 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:77LTd+pv0NNv6pRvjO25OtrXijU= 2024-04-09T23:24:41.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:77LTd+pv0NNv6pRvjO25OtrXijU='} 2024-04-09T23:24:41.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043b1a8dc7ac4f145-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:41.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:41.022 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:41.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:41.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:41.153 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:24:41.153 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-62 2024-04-09T23:24:41.157 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 5 2024-04-09T23:24:41.220 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.220 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.221 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.221 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.221 DEBUG:boto:Data: 2024-04-09T23:24:41.221 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.221 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.221 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.221 DEBUG:boto:Params: {} 2024-04-09T23:24:41.221 DEBUG:boto:Token: None 2024-04-09T23:24:41.221 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.222 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067d9e03502b7a116-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.225 DEBUG:boto:path=/xryqqk-62/obj17 2024-04-09T23:24:41.225 DEBUG:boto:auth_path=/xryqqk-62/obj17 2024-04-09T23:24:41.225 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.225 DEBUG:boto:Path: /xryqqk-62/obj17 2024-04-09T23:24:41.225 DEBUG:boto:Data: 2024-04-09T23:24:41.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.225 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.225 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.225 DEBUG:boto:Params: {} 2024-04-09T23:24:41.226 DEBUG:boto:Token: None 2024-04-09T23:24:41.226 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj17 2024-04-09T23:24:41.226 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:n4O5SGyz69/eKEbkpNpP/HEiqzg= 2024-04-09T23:24:41.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n4O5SGyz69/eKEbkpNpP/HEiqzg='} 2024-04-09T23:24:41.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e49e1a04fd07d115-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.234 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.234 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.234 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.234 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.234 DEBUG:boto:Data: 2024-04-09T23:24:41.234 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.234 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.234 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.234 DEBUG:boto:Params: {} 2024-04-09T23:24:41.234 DEBUG:boto:Token: None 2024-04-09T23:24:41.235 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.235 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9088f5dbdda0ab1-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.237 DEBUG:boto:path=/xryqqk-62/obj34 2024-04-09T23:24:41.237 DEBUG:boto:auth_path=/xryqqk-62/obj34 2024-04-09T23:24:41.237 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.237 DEBUG:boto:Path: /xryqqk-62/obj34 2024-04-09T23:24:41.237 DEBUG:boto:Data: 2024-04-09T23:24:41.237 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.237 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.238 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.238 DEBUG:boto:Params: {} 2024-04-09T23:24:41.238 DEBUG:boto:Token: None 2024-04-09T23:24:41.238 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj34 2024-04-09T23:24:41.238 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hrdzG4U3NrheRN9Hwen/AuZ32t4= 2024-04-09T23:24:41.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hrdzG4U3NrheRN9Hwen/AuZ32t4='} 2024-04-09T23:24:41.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b220d1e24907eed-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.283 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.283 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.283 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.283 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.283 DEBUG:boto:Data: 2024-04-09T23:24:41.283 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.283 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.284 DEBUG:boto:Params: {} 2024-04-09T23:24:41.284 DEBUG:boto:Token: None 2024-04-09T23:24:41.284 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.284 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.286 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd5d5ce7838b2770-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.287 DEBUG:boto:path=/xryqqk-62/obj51 2024-04-09T23:24:41.287 DEBUG:boto:auth_path=/xryqqk-62/obj51 2024-04-09T23:24:41.287 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.287 DEBUG:boto:Path: /xryqqk-62/obj51 2024-04-09T23:24:41.287 DEBUG:boto:Data: 2024-04-09T23:24:41.287 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.287 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.287 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.287 DEBUG:boto:Params: {} 2024-04-09T23:24:41.287 DEBUG:boto:Token: None 2024-04-09T23:24:41.288 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj51 2024-04-09T23:24:41.288 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/s7hAi9/tZFxSfIWaS/Rt3kw6no= 2024-04-09T23:24:41.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/s7hAi9/tZFxSfIWaS/Rt3kw6no='} 2024-04-09T23:24:41.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000848cc47cd93ed4fd-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.298 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.299 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.299 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.299 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.299 DEBUG:boto:Data: 2024-04-09T23:24:41.299 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.300 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.300 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.300 DEBUG:boto:Params: {} 2024-04-09T23:24:41.300 DEBUG:boto:Token: None 2024-04-09T23:24:41.301 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.301 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001185004c1f994d96-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.303 DEBUG:boto:path=/xryqqk-62/obj68 2024-04-09T23:24:41.304 DEBUG:boto:auth_path=/xryqqk-62/obj68 2024-04-09T23:24:41.304 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.304 DEBUG:boto:Path: /xryqqk-62/obj68 2024-04-09T23:24:41.304 DEBUG:boto:Data: 2024-04-09T23:24:41.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.304 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.304 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.304 DEBUG:boto:Params: {} 2024-04-09T23:24:41.304 DEBUG:boto:Token: None 2024-04-09T23:24:41.304 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj68 2024-04-09T23:24:41.305 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2zJ2Rfa16hJ/lnVTL5cduC4e6Dc= 2024-04-09T23:24:41.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2zJ2Rfa16hJ/lnVTL5cduC4e6Dc='} 2024-04-09T23:24:41.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063f7d4503749fde8-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.347 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.347 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.347 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.347 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.347 DEBUG:boto:Data: 2024-04-09T23:24:41.348 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.348 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.348 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.348 DEBUG:boto:Params: {} 2024-04-09T23:24:41.348 DEBUG:boto:Token: None 2024-04-09T23:24:41.348 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.348 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.351 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000713a63e99c031a61-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.351 DEBUG:boto:path=/xryqqk-62/obj85 2024-04-09T23:24:41.351 DEBUG:boto:auth_path=/xryqqk-62/obj85 2024-04-09T23:24:41.351 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.351 DEBUG:boto:Path: /xryqqk-62/obj85 2024-04-09T23:24:41.351 DEBUG:boto:Data: 2024-04-09T23:24:41.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.352 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.352 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.352 DEBUG:boto:Params: {} 2024-04-09T23:24:41.352 DEBUG:boto:Token: None 2024-04-09T23:24:41.352 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj85 2024-04-09T23:24:41.352 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:WIJQQb7G++3Lq04Qe4Vqhl6ZpQE= 2024-04-09T23:24:41.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WIJQQb7G++3Lq04Qe4Vqhl6ZpQE='} 2024-04-09T23:24:41.360 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000720acebcfeec36da-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:41.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:41.360 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:41.413 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:41.473 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:24:41.474 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-62 2024-04-09T23:24:41.480 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 10 2024-04-09T23:24:41.556 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.556 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.556 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.557 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.557 DEBUG:boto:Data: 2024-04-09T23:24:41.557 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.557 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.557 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.557 DEBUG:boto:Params: {} 2024-04-09T23:24:41.557 DEBUG:boto:Token: None 2024-04-09T23:24:41.557 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.558 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7d2bece1a91ef00-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.561 DEBUG:boto:path=/xryqqk-62/obj19 2024-04-09T23:24:41.561 DEBUG:boto:auth_path=/xryqqk-62/obj19 2024-04-09T23:24:41.561 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.561 DEBUG:boto:Path: /xryqqk-62/obj19 2024-04-09T23:24:41.561 DEBUG:boto:Data: 2024-04-09T23:24:41.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.561 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.561 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.561 DEBUG:boto:Params: {} 2024-04-09T23:24:41.561 DEBUG:boto:Token: None 2024-04-09T23:24:41.562 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj19 2024-04-09T23:24:41.562 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r5vxXGaUYx8HTpAnPizIDh3FZo4= 2024-04-09T23:24:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r5vxXGaUYx8HTpAnPizIDh3FZo4='} 2024-04-09T23:24:41.576 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fd2e4b31dff5f46-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.576 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.577 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.577 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.577 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.577 DEBUG:boto:Data: 2024-04-09T23:24:41.577 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.577 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.577 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.577 DEBUG:boto:Params: {} 2024-04-09T23:24:41.577 DEBUG:boto:Token: None 2024-04-09T23:24:41.578 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.578 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f038931c63a5e447-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.580 DEBUG:boto:path=/xryqqk-62/obj38 2024-04-09T23:24:41.580 DEBUG:boto:auth_path=/xryqqk-62/obj38 2024-04-09T23:24:41.580 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.581 DEBUG:boto:Path: /xryqqk-62/obj38 2024-04-09T23:24:41.581 DEBUG:boto:Data: 2024-04-09T23:24:41.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.581 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.581 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.581 DEBUG:boto:Params: {} 2024-04-09T23:24:41.581 DEBUG:boto:Token: None 2024-04-09T23:24:41.581 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj38 2024-04-09T23:24:41.581 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cl6TJypExdqdghWxguro/5ZfeOk= 2024-04-09T23:24:41.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cl6TJypExdqdghWxguro/5ZfeOk='} 2024-04-09T23:24:41.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009194b2952c603ef0-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.631 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.631 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.631 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.631 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.631 DEBUG:boto:Data: 2024-04-09T23:24:41.631 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.631 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.631 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.632 DEBUG:boto:Params: {} 2024-04-09T23:24:41.632 DEBUG:boto:Token: None 2024-04-09T23:24:41.632 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.632 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c033fdeb29f9fc91-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.635 DEBUG:boto:path=/xryqqk-62/obj57 2024-04-09T23:24:41.635 DEBUG:boto:auth_path=/xryqqk-62/obj57 2024-04-09T23:24:41.635 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.635 DEBUG:boto:Path: /xryqqk-62/obj57 2024-04-09T23:24:41.635 DEBUG:boto:Data: 2024-04-09T23:24:41.635 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.635 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.636 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.636 DEBUG:boto:Params: {} 2024-04-09T23:24:41.636 DEBUG:boto:Token: None 2024-04-09T23:24:41.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj57 2024-04-09T23:24:41.636 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7A+wfCpyDoAvc1O+2SHB9v876EA= 2024-04-09T23:24:41.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7A+wfCpyDoAvc1O+2SHB9v876EA='} 2024-04-09T23:24:41.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000551e6235a70b21de-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.640 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.640 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.640 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.640 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.641 DEBUG:boto:Data: 2024-04-09T23:24:41.641 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.641 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.641 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.641 DEBUG:boto:Params: {} 2024-04-09T23:24:41.641 DEBUG:boto:Token: None 2024-04-09T23:24:41.641 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.641 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042c940f0ca69221e-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.644 DEBUG:boto:path=/xryqqk-62/obj76 2024-04-09T23:24:41.644 DEBUG:boto:auth_path=/xryqqk-62/obj76 2024-04-09T23:24:41.644 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.644 DEBUG:boto:Path: /xryqqk-62/obj76 2024-04-09T23:24:41.644 DEBUG:boto:Data: 2024-04-09T23:24:41.645 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.645 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.645 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.645 DEBUG:boto:Params: {} 2024-04-09T23:24:41.645 DEBUG:boto:Token: None 2024-04-09T23:24:41.645 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj76 2024-04-09T23:24:41.645 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/WycllqdqPbwyGJlvEUCegAlgWs= 2024-04-09T23:24:41.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/WycllqdqPbwyGJlvEUCegAlgWs='} 2024-04-09T23:24:41.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008da02ddd67c7f757-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.687 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.687 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.687 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.687 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.687 DEBUG:boto:Data: 2024-04-09T23:24:41.687 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.687 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.688 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.688 DEBUG:boto:Params: {} 2024-04-09T23:24:41.688 DEBUG:boto:Token: None 2024-04-09T23:24:41.688 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.688 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.691 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092325ffc09b01458-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.691 DEBUG:boto:path=/xryqqk-62/obj95 2024-04-09T23:24:41.691 DEBUG:boto:auth_path=/xryqqk-62/obj95 2024-04-09T23:24:41.691 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.691 DEBUG:boto:Path: /xryqqk-62/obj95 2024-04-09T23:24:41.691 DEBUG:boto:Data: 2024-04-09T23:24:41.692 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.692 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.692 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.692 DEBUG:boto:Params: {} 2024-04-09T23:24:41.692 DEBUG:boto:Token: None 2024-04-09T23:24:41.692 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj95 2024-04-09T23:24:41.693 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4jb8naZFiSkag03GHfanb9x9dco= 2024-04-09T23:24:41.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4jb8naZFiSkag03GHfanb9x9dco='} 2024-04-09T23:24:41.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1698b54ead1dacd-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:41.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:41.701 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:41.755 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:41.755 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:24:41.823 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-62 2024-04-09T23:24:41.830 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 15 2024-04-09T23:24:41.911 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.911 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.911 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.911 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.911 DEBUG:boto:Data: 2024-04-09T23:24:41.911 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.911 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.912 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.912 DEBUG:boto:Params: {} 2024-04-09T23:24:41.912 DEBUG:boto:Token: None 2024-04-09T23:24:41.912 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.912 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033bbbc7f85ba73b7-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.915 DEBUG:boto:path=/xryqqk-62/obj23 2024-04-09T23:24:41.915 DEBUG:boto:auth_path=/xryqqk-62/obj23 2024-04-09T23:24:41.915 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.916 DEBUG:boto:Path: /xryqqk-62/obj23 2024-04-09T23:24:41.916 DEBUG:boto:Data: 2024-04-09T23:24:41.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.916 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.916 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.916 DEBUG:boto:Params: {} 2024-04-09T23:24:41.916 DEBUG:boto:Token: None 2024-04-09T23:24:41.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj23 2024-04-09T23:24:41.916 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QK3DsefEOZ+gMKYvSiJaoweaXJM= 2024-04-09T23:24:41.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QK3DsefEOZ+gMKYvSiJaoweaXJM='} 2024-04-09T23:24:41.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038232fab3b9146f4-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.963 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.963 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.963 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.963 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.963 DEBUG:boto:Data: 2024-04-09T23:24:41.963 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.963 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.963 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.963 DEBUG:boto:Params: {} 2024-04-09T23:24:41.963 DEBUG:boto:Token: None 2024-04-09T23:24:41.964 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.964 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006843fa4decbdf7bd-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.968 DEBUG:boto:path=/xryqqk-62/obj46 2024-04-09T23:24:41.968 DEBUG:boto:auth_path=/xryqqk-62/obj46 2024-04-09T23:24:41.968 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.968 DEBUG:boto:Path: /xryqqk-62/obj46 2024-04-09T23:24:41.968 DEBUG:boto:Data: 2024-04-09T23:24:41.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.968 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.968 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.968 DEBUG:boto:Params: {} 2024-04-09T23:24:41.969 DEBUG:boto:Token: None 2024-04-09T23:24:41.969 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj46 2024-04-09T23:24:41.969 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bbSJNNDQL9w6NbKr8N1SLVoFUTQ= 2024-04-09T23:24:41.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bbSJNNDQL9w6NbKr8N1SLVoFUTQ='} 2024-04-09T23:24:41.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004e11e579b99d272-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.981 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:41.981 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:41.981 DEBUG:boto:Method: HEAD 2024-04-09T23:24:41.981 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:41.981 DEBUG:boto:Data: 2024-04-09T23:24:41.981 DEBUG:boto:Headers: {} 2024-04-09T23:24:41.981 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.981 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.981 DEBUG:boto:Params: {} 2024-04-09T23:24:41.981 DEBUG:boto:Token: None 2024-04-09T23:24:41.982 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/ 2024-04-09T23:24:41.982 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-09T23:24:41.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-09T23:24:41.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8adf1e3051272f3-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:41.985 DEBUG:boto:path=/xryqqk-62/obj69 2024-04-09T23:24:41.985 DEBUG:boto:auth_path=/xryqqk-62/obj69 2024-04-09T23:24:41.985 DEBUG:boto:Method: PUT 2024-04-09T23:24:41.985 DEBUG:boto:Path: /xryqqk-62/obj69 2024-04-09T23:24:41.985 DEBUG:boto:Data: 2024-04-09T23:24:41.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:41.985 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:41.985 DEBUG:boto:Port: 8000 2024-04-09T23:24:41.985 DEBUG:boto:Params: {} 2024-04-09T23:24:41.985 DEBUG:boto:Token: None 2024-04-09T23:24:41.986 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:41 GMT /xryqqk-62/obj69 2024-04-09T23:24:41.986 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qBzLxAkDk7WlZXrexzLfjfgsbkc= 2024-04-09T23:24:41.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qBzLxAkDk7WlZXrexzLfjfgsbkc='} 2024-04-09T23:24:42.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000414119d22d09363b-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.031 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.031 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.031 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.031 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.031 DEBUG:boto:Data: 2024-04-09T23:24:42.031 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.031 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.031 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.031 DEBUG:boto:Params: {} 2024-04-09T23:24:42.031 DEBUG:boto:Token: None 2024-04-09T23:24:42.032 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.032 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.034 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2a34559cfb23081-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.035 DEBUG:boto:path=/xryqqk-62/obj92 2024-04-09T23:24:42.035 DEBUG:boto:auth_path=/xryqqk-62/obj92 2024-04-09T23:24:42.035 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.035 DEBUG:boto:Path: /xryqqk-62/obj92 2024-04-09T23:24:42.035 DEBUG:boto:Data: 2024-04-09T23:24:42.035 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.035 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.035 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.035 DEBUG:boto:Params: {} 2024-04-09T23:24:42.035 DEBUG:boto:Token: None 2024-04-09T23:24:42.036 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj92 2024-04-09T23:24:42.036 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dV6V9xvMT8dgG18tmFeZvTPwzTQ= 2024-04-09T23:24:42.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dV6V9xvMT8dgG18tmFeZvTPwzTQ='} 2024-04-09T23:24:42.044 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091a9d8bc05033b4c-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.044 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.044 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.044 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.044 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.044 DEBUG:boto:Data: 2024-04-09T23:24:42.044 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.044 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.044 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.044 DEBUG:boto:Params: {} 2024-04-09T23:24:42.045 DEBUG:boto:Token: None 2024-04-09T23:24:42.045 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.045 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.047 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019f9228bfeb1fcc2-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.047 DEBUG:boto:path=/xryqqk-62/obj115 2024-04-09T23:24:42.048 DEBUG:boto:auth_path=/xryqqk-62/obj115 2024-04-09T23:24:42.048 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.048 DEBUG:boto:Path: /xryqqk-62/obj115 2024-04-09T23:24:42.048 DEBUG:boto:Data: 2024-04-09T23:24:42.048 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.048 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.048 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.048 DEBUG:boto:Params: {} 2024-04-09T23:24:42.048 DEBUG:boto:Token: None 2024-04-09T23:24:42.048 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj115 2024-04-09T23:24:42.049 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:x/KA0lAxxC/Y+sVELQYRPf/aaV0= 2024-04-09T23:24:42.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x/KA0lAxxC/Y+sVELQYRPf/aaV0='} 2024-04-09T23:24:42.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c887604b3d8225e-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:42.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:42.091 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:42.146 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:42.210 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:24:42.210 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-62 2024-04-09T23:24:42.219 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 20 2024-04-09T23:24:42.290 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.291 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.291 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.291 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.291 DEBUG:boto:Data: 2024-04-09T23:24:42.291 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.291 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.291 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.291 DEBUG:boto:Params: {} 2024-04-09T23:24:42.291 DEBUG:boto:Token: None 2024-04-09T23:24:42.292 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.292 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5c2f0abcb845d59-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.295 DEBUG:boto:path=/xryqqk-62/obj29 2024-04-09T23:24:42.295 DEBUG:boto:auth_path=/xryqqk-62/obj29 2024-04-09T23:24:42.295 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.296 DEBUG:boto:Path: /xryqqk-62/obj29 2024-04-09T23:24:42.296 DEBUG:boto:Data: 2024-04-09T23:24:42.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.296 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.296 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.296 DEBUG:boto:Params: {} 2024-04-09T23:24:42.296 DEBUG:boto:Token: None 2024-04-09T23:24:42.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj29 2024-04-09T23:24:42.296 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:r/JeDXNTypn6Vy67TZ8e5Ju1jiM= 2024-04-09T23:24:42.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r/JeDXNTypn6Vy67TZ8e5Ju1jiM='} 2024-04-09T23:24:42.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e0ac59af23c32bd-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.303 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.303 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.303 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.303 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.303 DEBUG:boto:Data: 2024-04-09T23:24:42.303 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.303 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.303 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.303 DEBUG:boto:Params: {} 2024-04-09T23:24:42.304 DEBUG:boto:Token: None 2024-04-09T23:24:42.304 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.304 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e95cb797e6e6bd4-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.307 DEBUG:boto:path=/xryqqk-62/obj58 2024-04-09T23:24:42.307 DEBUG:boto:auth_path=/xryqqk-62/obj58 2024-04-09T23:24:42.307 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.307 DEBUG:boto:Path: /xryqqk-62/obj58 2024-04-09T23:24:42.308 DEBUG:boto:Data: 2024-04-09T23:24:42.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.308 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.308 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.308 DEBUG:boto:Params: {} 2024-04-09T23:24:42.308 DEBUG:boto:Token: None 2024-04-09T23:24:42.308 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj58 2024-04-09T23:24:42.308 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EyoLJHbQqPoJoiAWlq0qUR2PISo= 2024-04-09T23:24:42.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EyoLJHbQqPoJoiAWlq0qUR2PISo='} 2024-04-09T23:24:42.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea0edede445bc863-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.351 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.351 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.351 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.351 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.351 DEBUG:boto:Data: 2024-04-09T23:24:42.351 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.351 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.351 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.351 DEBUG:boto:Params: {} 2024-04-09T23:24:42.352 DEBUG:boto:Token: None 2024-04-09T23:24:42.352 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.352 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6ff1cda1185419f-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.355 DEBUG:boto:path=/xryqqk-62/obj87 2024-04-09T23:24:42.355 DEBUG:boto:auth_path=/xryqqk-62/obj87 2024-04-09T23:24:42.355 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.355 DEBUG:boto:Path: /xryqqk-62/obj87 2024-04-09T23:24:42.356 DEBUG:boto:Data: 2024-04-09T23:24:42.356 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.356 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.356 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.356 DEBUG:boto:Params: {} 2024-04-09T23:24:42.356 DEBUG:boto:Token: None 2024-04-09T23:24:42.356 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj87 2024-04-09T23:24:42.356 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ki0zpER0Nsq76EOZ1jtiZlH5ZNQ= 2024-04-09T23:24:42.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ki0zpER0Nsq76EOZ1jtiZlH5ZNQ='} 2024-04-09T23:24:42.365 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082d910e2e6651aba-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.365 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.365 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.365 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.365 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.366 DEBUG:boto:Data: 2024-04-09T23:24:42.366 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.366 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.366 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.366 DEBUG:boto:Params: {} 2024-04-09T23:24:42.366 DEBUG:boto:Token: None 2024-04-09T23:24:42.366 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.366 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000124563e84a2f2f19-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.370 DEBUG:boto:path=/xryqqk-62/obj116 2024-04-09T23:24:42.370 DEBUG:boto:auth_path=/xryqqk-62/obj116 2024-04-09T23:24:42.370 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.370 DEBUG:boto:Path: /xryqqk-62/obj116 2024-04-09T23:24:42.370 DEBUG:boto:Data: 2024-04-09T23:24:42.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.370 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.370 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.370 DEBUG:boto:Params: {} 2024-04-09T23:24:42.371 DEBUG:boto:Token: None 2024-04-09T23:24:42.371 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj116 2024-04-09T23:24:42.371 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:reoom6f+AY+tXslaWxj9PSfyJU8= 2024-04-09T23:24:42.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:reoom6f+AY+tXslaWxj9PSfyJU8='} 2024-04-09T23:24:42.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ddbad886b4d52d2-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.415 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.415 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.415 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.415 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.415 DEBUG:boto:Data: 2024-04-09T23:24:42.415 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.415 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.415 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.415 DEBUG:boto:Params: {} 2024-04-09T23:24:42.416 DEBUG:boto:Token: None 2024-04-09T23:24:42.416 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.416 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d52caa1b79f867f-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.419 DEBUG:boto:path=/xryqqk-62/obj145 2024-04-09T23:24:42.419 DEBUG:boto:auth_path=/xryqqk-62/obj145 2024-04-09T23:24:42.419 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.419 DEBUG:boto:Path: /xryqqk-62/obj145 2024-04-09T23:24:42.419 DEBUG:boto:Data: 2024-04-09T23:24:42.419 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.419 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.420 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.420 DEBUG:boto:Params: {} 2024-04-09T23:24:42.420 DEBUG:boto:Token: None 2024-04-09T23:24:42.420 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj145 2024-04-09T23:24:42.420 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:T+dIcypfxJNiKV9uXFGYFK0u/Ms= 2024-04-09T23:24:42.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T+dIcypfxJNiKV9uXFGYFK0u/Ms='} 2024-04-09T23:24:42.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b760b021fc17191d-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:42.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:42.426 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:42.478 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:42.479 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:42.531 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:24:42.532 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-62 2024-04-09T23:24:42.542 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 25 2024-04-09T23:24:42.617 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.617 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.617 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.617 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.617 DEBUG:boto:Data: 2024-04-09T23:24:42.617 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.617 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.618 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.618 DEBUG:boto:Params: {} 2024-04-09T23:24:42.618 DEBUG:boto:Token: None 2024-04-09T23:24:42.618 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.618 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.622 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017c6f03e2c71c79b-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.622 DEBUG:boto:path=/xryqqk-62/obj31 2024-04-09T23:24:42.622 DEBUG:boto:auth_path=/xryqqk-62/obj31 2024-04-09T23:24:42.622 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.622 DEBUG:boto:Path: /xryqqk-62/obj31 2024-04-09T23:24:42.622 DEBUG:boto:Data: 2024-04-09T23:24:42.623 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.623 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.623 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.623 DEBUG:boto:Params: {} 2024-04-09T23:24:42.623 DEBUG:boto:Token: None 2024-04-09T23:24:42.623 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj31 2024-04-09T23:24:42.623 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:HbxIG8Y7PB2LydBX4xCp7tM/a70= 2024-04-09T23:24:42.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HbxIG8Y7PB2LydBX4xCp7tM/a70='} 2024-04-09T23:24:42.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b687462dfc2b891-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.634 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.635 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.635 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.635 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.635 DEBUG:boto:Data: 2024-04-09T23:24:42.635 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.635 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.635 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.635 DEBUG:boto:Params: {} 2024-04-09T23:24:42.635 DEBUG:boto:Token: None 2024-04-09T23:24:42.635 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.636 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcf61b548f5a155-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.639 DEBUG:boto:path=/xryqqk-62/obj62 2024-04-09T23:24:42.639 DEBUG:boto:auth_path=/xryqqk-62/obj62 2024-04-09T23:24:42.639 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.639 DEBUG:boto:Path: /xryqqk-62/obj62 2024-04-09T23:24:42.640 DEBUG:boto:Data: 2024-04-09T23:24:42.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.640 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.640 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.640 DEBUG:boto:Params: {} 2024-04-09T23:24:42.640 DEBUG:boto:Token: None 2024-04-09T23:24:42.640 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj62 2024-04-09T23:24:42.640 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TpsfjG146XAC4Tm49nPRVIlYxdw= 2024-04-09T23:24:42.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TpsfjG146XAC4Tm49nPRVIlYxdw='} 2024-04-09T23:24:42.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d94f128c47610d8-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.683 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.683 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.683 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.683 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.683 DEBUG:boto:Data: 2024-04-09T23:24:42.683 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.684 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.684 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.684 DEBUG:boto:Params: {} 2024-04-09T23:24:42.684 DEBUG:boto:Token: None 2024-04-09T23:24:42.684 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.684 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae0e1e6109a32ac3-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.687 DEBUG:boto:path=/xryqqk-62/obj93 2024-04-09T23:24:42.687 DEBUG:boto:auth_path=/xryqqk-62/obj93 2024-04-09T23:24:42.688 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.688 DEBUG:boto:Path: /xryqqk-62/obj93 2024-04-09T23:24:42.688 DEBUG:boto:Data: 2024-04-09T23:24:42.688 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.688 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.688 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.688 DEBUG:boto:Params: {} 2024-04-09T23:24:42.688 DEBUG:boto:Token: None 2024-04-09T23:24:42.689 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj93 2024-04-09T23:24:42.689 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:sirb4clRUNlQLg4HQ1poRmkV8UQ= 2024-04-09T23:24:42.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sirb4clRUNlQLg4HQ1poRmkV8UQ='} 2024-04-09T23:24:42.700 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056d218329b1e35bf-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.700 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.700 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.701 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.701 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.701 DEBUG:boto:Data: 2024-04-09T23:24:42.701 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.701 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.701 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.701 DEBUG:boto:Params: {} 2024-04-09T23:24:42.701 DEBUG:boto:Token: None 2024-04-09T23:24:42.701 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.702 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.705 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8e38ae761de6ed9-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.705 DEBUG:boto:path=/xryqqk-62/obj124 2024-04-09T23:24:42.705 DEBUG:boto:auth_path=/xryqqk-62/obj124 2024-04-09T23:24:42.705 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.705 DEBUG:boto:Path: /xryqqk-62/obj124 2024-04-09T23:24:42.705 DEBUG:boto:Data: 2024-04-09T23:24:42.706 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.706 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.706 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.706 DEBUG:boto:Params: {} 2024-04-09T23:24:42.706 DEBUG:boto:Token: None 2024-04-09T23:24:42.706 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj124 2024-04-09T23:24:42.706 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qAzgQq2TKZNcllHHqoExcPQ4tb0= 2024-04-09T23:24:42.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qAzgQq2TKZNcllHHqoExcPQ4tb0='} 2024-04-09T23:24:42.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c73ba4ca1c945d3d-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.755 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:42.755 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:42.755 DEBUG:boto:Method: HEAD 2024-04-09T23:24:42.755 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:42.755 DEBUG:boto:Data: 2024-04-09T23:24:42.755 DEBUG:boto:Headers: {} 2024-04-09T23:24:42.755 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.755 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.755 DEBUG:boto:Params: {} 2024-04-09T23:24:42.756 DEBUG:boto:Token: None 2024-04-09T23:24:42.756 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/ 2024-04-09T23:24:42.756 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-09T23:24:42.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-09T23:24:42.759 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c601c45c272d15-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.760 DEBUG:boto:path=/xryqqk-62/obj155 2024-04-09T23:24:42.760 DEBUG:boto:auth_path=/xryqqk-62/obj155 2024-04-09T23:24:42.760 DEBUG:boto:Method: PUT 2024-04-09T23:24:42.760 DEBUG:boto:Path: /xryqqk-62/obj155 2024-04-09T23:24:42.760 DEBUG:boto:Data: 2024-04-09T23:24:42.760 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:42.760 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:42.760 DEBUG:boto:Port: 8000 2024-04-09T23:24:42.760 DEBUG:boto:Params: {} 2024-04-09T23:24:42.760 DEBUG:boto:Token: None 2024-04-09T23:24:42.761 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:42 GMT /xryqqk-62/obj155 2024-04-09T23:24:42.761 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:EnEOVB8H6DWIcHZwZbdmVyXsS0o= 2024-04-09T23:24:42.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EnEOVB8H6DWIcHZwZbdmVyXsS0o='} 2024-04-09T23:24:42.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f20d768cdb960ee-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:42.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:42.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:42.768 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:42.821 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:42.821 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:42.888 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:24:42.888 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-62 2024-04-09T23:24:42.900 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 30 2024-04-09T23:24:43.006 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:43.007 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:43.007 DEBUG:boto:Method: HEAD 2024-04-09T23:24:43.007 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:43.007 DEBUG:boto:Data: 2024-04-09T23:24:43.007 DEBUG:boto:Headers: {} 2024-04-09T23:24:43.007 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.007 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.007 DEBUG:boto:Params: {} 2024-04-09T23:24:43.007 DEBUG:boto:Token: None 2024-04-09T23:24:43.008 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/ 2024-04-09T23:24:43.008 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-09T23:24:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-09T23:24:43.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a87fc49616a3f43-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.012 DEBUG:boto:path=/xryqqk-62/obj37 2024-04-09T23:24:43.012 DEBUG:boto:auth_path=/xryqqk-62/obj37 2024-04-09T23:24:43.012 DEBUG:boto:Method: PUT 2024-04-09T23:24:43.012 DEBUG:boto:Path: /xryqqk-62/obj37 2024-04-09T23:24:43.012 DEBUG:boto:Data: 2024-04-09T23:24:43.012 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:43.013 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.013 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.013 DEBUG:boto:Params: {} 2024-04-09T23:24:43.013 DEBUG:boto:Token: None 2024-04-09T23:24:43.013 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/obj37 2024-04-09T23:24:43.013 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SlvWFtEF01uE+ftpbeENub0eNdQ= 2024-04-09T23:24:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SlvWFtEF01uE+ftpbeENub0eNdQ='} 2024-04-09T23:24:43.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a9debe9216021f4-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.059 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:43.059 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:43.059 DEBUG:boto:Method: HEAD 2024-04-09T23:24:43.059 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:43.059 DEBUG:boto:Data: 2024-04-09T23:24:43.059 DEBUG:boto:Headers: {} 2024-04-09T23:24:43.059 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.059 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.059 DEBUG:boto:Params: {} 2024-04-09T23:24:43.060 DEBUG:boto:Token: None 2024-04-09T23:24:43.060 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/ 2024-04-09T23:24:43.060 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-09T23:24:43.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-09T23:24:43.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008417dece65ec2494-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.064 DEBUG:boto:path=/xryqqk-62/obj74 2024-04-09T23:24:43.064 DEBUG:boto:auth_path=/xryqqk-62/obj74 2024-04-09T23:24:43.064 DEBUG:boto:Method: PUT 2024-04-09T23:24:43.064 DEBUG:boto:Path: /xryqqk-62/obj74 2024-04-09T23:24:43.064 DEBUG:boto:Data: 2024-04-09T23:24:43.064 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:43.064 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.064 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.064 DEBUG:boto:Params: {} 2024-04-09T23:24:43.064 DEBUG:boto:Token: None 2024-04-09T23:24:43.065 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/obj74 2024-04-09T23:24:43.065 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:db7Pm8ncuVP/p/h+URq4Mkxev6o= 2024-04-09T23:24:43.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:db7Pm8ncuVP/p/h+URq4Mkxev6o='} 2024-04-09T23:24:43.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000386233700cb5eb81-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.073 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:43.073 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:43.073 DEBUG:boto:Method: HEAD 2024-04-09T23:24:43.074 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:43.074 DEBUG:boto:Data: 2024-04-09T23:24:43.074 DEBUG:boto:Headers: {} 2024-04-09T23:24:43.074 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.074 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.074 DEBUG:boto:Params: {} 2024-04-09T23:24:43.074 DEBUG:boto:Token: None 2024-04-09T23:24:43.074 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/ 2024-04-09T23:24:43.074 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-09T23:24:43.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-09T23:24:43.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000692b7c9927560358-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.078 DEBUG:boto:path=/xryqqk-62/obj111 2024-04-09T23:24:43.079 DEBUG:boto:auth_path=/xryqqk-62/obj111 2024-04-09T23:24:43.079 DEBUG:boto:Method: PUT 2024-04-09T23:24:43.079 DEBUG:boto:Path: /xryqqk-62/obj111 2024-04-09T23:24:43.079 DEBUG:boto:Data: 2024-04-09T23:24:43.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:43.079 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.079 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.079 DEBUG:boto:Params: {} 2024-04-09T23:24:43.079 DEBUG:boto:Token: None 2024-04-09T23:24:43.079 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/obj111 2024-04-09T23:24:43.080 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dC30iN+CdgoZ2f+2cnRRIb0L30k= 2024-04-09T23:24:43.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dC30iN+CdgoZ2f+2cnRRIb0L30k='} 2024-04-09T23:24:43.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd5c7aaf8f875ce5-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.123 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:43.123 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:43.123 DEBUG:boto:Method: HEAD 2024-04-09T23:24:43.123 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:43.123 DEBUG:boto:Data: 2024-04-09T23:24:43.123 DEBUG:boto:Headers: {} 2024-04-09T23:24:43.123 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.123 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.123 DEBUG:boto:Params: {} 2024-04-09T23:24:43.124 DEBUG:boto:Token: None 2024-04-09T23:24:43.124 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/ 2024-04-09T23:24:43.124 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-09T23:24:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-09T23:24:43.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015fb92bc16cef68a-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.128 DEBUG:boto:path=/xryqqk-62/obj148 2024-04-09T23:24:43.128 DEBUG:boto:auth_path=/xryqqk-62/obj148 2024-04-09T23:24:43.128 DEBUG:boto:Method: PUT 2024-04-09T23:24:43.128 DEBUG:boto:Path: /xryqqk-62/obj148 2024-04-09T23:24:43.128 DEBUG:boto:Data: 2024-04-09T23:24:43.128 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:43.128 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.129 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.129 DEBUG:boto:Params: {} 2024-04-09T23:24:43.129 DEBUG:boto:Token: None 2024-04-09T23:24:43.129 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/obj148 2024-04-09T23:24:43.129 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:holrHiDaYjTebevyoAZfABRSNP8= 2024-04-09T23:24:43.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:holrHiDaYjTebevyoAZfABRSNP8='} 2024-04-09T23:24:43.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bccbb30ccf84379a-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.138 DEBUG:boto:path=/xryqqk-62/ 2024-04-09T23:24:43.138 DEBUG:boto:auth_path=/xryqqk-62/ 2024-04-09T23:24:43.138 DEBUG:boto:Method: HEAD 2024-04-09T23:24:43.138 DEBUG:boto:Path: /xryqqk-62/ 2024-04-09T23:24:43.139 DEBUG:boto:Data: 2024-04-09T23:24:43.139 DEBUG:boto:Headers: {} 2024-04-09T23:24:43.139 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.139 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.139 DEBUG:boto:Params: {} 2024-04-09T23:24:43.139 DEBUG:boto:Token: None 2024-04-09T23:24:43.139 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/ 2024-04-09T23:24:43.139 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-09T23:24:43.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-09T23:24:43.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6efc7fb54a8ee28-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.143 DEBUG:boto:path=/xryqqk-62/obj185 2024-04-09T23:24:43.143 DEBUG:boto:auth_path=/xryqqk-62/obj185 2024-04-09T23:24:43.143 DEBUG:boto:Method: PUT 2024-04-09T23:24:43.143 DEBUG:boto:Path: /xryqqk-62/obj185 2024-04-09T23:24:43.143 DEBUG:boto:Data: 2024-04-09T23:24:43.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:24:43.143 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:24:43.143 DEBUG:boto:Port: 8000 2024-04-09T23:24:43.144 DEBUG:boto:Params: {} 2024-04-09T23:24:43.144 DEBUG:boto:Token: None 2024-04-09T23:24:43.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:24:43 GMT /xryqqk-62/obj185 2024-04-09T23:24:43.144 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UpgfzssyV53lcI26dGozEc6Xv3U= 2024-04-09T23:24:43.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UpgfzssyV53lcI26dGozEc6Xv3U='} 2024-04-09T23:24:43.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081441446340ffe5e-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:24:43.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:43.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:24:43.191 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:24:43.242 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:43.242 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:43.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:43.279+0000 7ff672c7f780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-09T23:24:43.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:43.280+0000 7ff672c7f780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-09T23:24:43.304 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-09T23:24:43.305 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-09T23:24:43.307 INFO:tasks.rgw.c1.client.1:Started 2024-04-09T23:24:43.307 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T23:24:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-09T23:24:43.333 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T23:24:43.333 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T23:24:43.333 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-09T23:24:44.334 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2024-04-09T23:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-09T23:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2024-04-09T23:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-09T23:24:44.362 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2024-04-09T23:24:44.362 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-62 2024-04-09T23:24:44.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:24:44.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:24:44.363 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:24:44.460 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:44.460 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:44.504 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-62 2024-04-09T23:24:44.519 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-62 2024-04-09T23:24:44.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:24:44.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:24:44.519 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:24:44.575 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:24:44.575 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:24:44.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:44.621+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:24:45.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-09T23:24:49.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:49.621+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:24:54.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:54.622+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:24:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:24:59.622+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:04.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:04.623+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:09.624+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:14.624+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:19.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:19.625+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:24.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:24.625+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:29.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:29.626+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:34.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:34.627+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:39.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:39.628+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:44.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:44.627+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:49.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:49.628+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:54.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:54.628+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:25:59.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:25:59.629+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:04.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:04.630+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:09.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:09.630+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:14.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:14.631+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:19.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:19.632+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:24.632+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:29.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:29.633+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:34.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:34.634+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:39.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:39.634+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:44.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:44.635+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:49.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:49.635+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:54.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:54.635+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:26:59.636+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:04.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:04.637+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:09.637+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:14.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:14.638+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:19.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:19.639+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:24.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:24.639+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:29.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:29.639+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:34.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:34.640+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:39.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:39.641+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:44.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:44.641+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:49.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:49.642+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:54.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:54.642+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:27:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:27:59.643+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:04.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:04.644+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:09.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:09.645+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:14.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:14.645+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:19.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:19.646+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:24.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:24.647+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:29.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:29.647+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:34.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:34.647+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:39.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:39.648+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:44.649+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:49.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:49.650+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:54.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:54.651+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:28:59.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:28:59.650+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:04.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:04.651+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:09.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:09.652+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:14.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:14.652+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:19.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:19.653+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:24.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:24.653+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:29.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:29.654+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:34.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:34.654+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-09T23:29:39.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:39.655+0000 7f5019b36780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:29:39.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:39.655+0000 7f5019b36780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:29:39.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:39.655+0000 7f5019b36780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:29:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:29:39.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... FAIL 2024-04-09T23:29:39.677 DEBUG:boto:path=/xryqqk-63/ 2024-04-09T23:29:39.677 DEBUG:boto:auth_path=/xryqqk-63/ 2024-04-09T23:29:39.677 DEBUG:boto:Method: PUT 2024-04-09T23:29:39.677 DEBUG:boto:Path: /xryqqk-63/ 2024-04-09T23:29:39.677 DEBUG:boto:Data: 2024-04-09T23:29:39.677 DEBUG:boto:Headers: {} 2024-04-09T23:29:39.677 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:29:39.678 DEBUG:boto:Port: 8000 2024-04-09T23:29:39.678 DEBUG:boto:Params: {} 2024-04-09T23:29:39.678 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:29:39.678 DEBUG:boto:Token: None 2024-04-09T23:29:39.679 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:29:39 GMT /xryqqk-63/ 2024-04-09T23:29:39.679 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Z3XOUPghnUH5qXx89WD/RPZwC3A= 2024-04-09T23:29:39.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:29:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Z3XOUPghnUH5qXx89WD/RPZwC3A=', 'Content-Length': '0'} 2024-04-09T23:29:39.701 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002e13d850796e6b54-006615cf63-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:29:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:29:39.701 DEBUG:rgw_multi.tests:created bucket=xryqqk-63 2024-04-09T23:29:39.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:29:39.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:29:39.702 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:29:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:29:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:29:39.811 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:29:39.829 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705379.699621_1231.1", 2024-04-09T23:29:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:29:39.699621Z" 2024-04-09T23:29:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:29:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:29:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:29:39.831 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:29:39.832 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705379.699621_1231.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:29:39.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:29:39.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:29:39.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:29:39.833 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:29:39.884 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:29:39.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:29:39.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:39.933+0000 7f869af6d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:29:39.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:39.934+0000 7f869af6d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:29:39.949 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:29:39.949 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:29:39.949 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:29:39.950 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:29:39.951 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:29:39.952 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705379.699621_1231.1", 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:29:39.699621Z", 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:29:39.953 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:29:39.954 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:29:39Z" 2024-04-09T23:29:39.955 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:29:39.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705379.699621_1231.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:29:39.699621Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:29:39Z" } 2024-04-09T23:29:39.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705379.699621_1231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:29:39.699621Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:29:39.956 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705379.699621_1231.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:29:39.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705379.699621_1231.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:29:39.957 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:29:39.957 DEBUG:boto:path=/xryqqk-63/ 2024-04-09T23:29:39.957 DEBUG:boto:auth_path=/xryqqk-63/ 2024-04-09T23:29:39.957 DEBUG:boto:Method: HEAD 2024-04-09T23:29:39.957 DEBUG:boto:Path: /xryqqk-63/ 2024-04-09T23:29:39.957 DEBUG:boto:Data: 2024-04-09T23:29:39.957 DEBUG:boto:Headers: {} 2024-04-09T23:29:39.957 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:29:39.957 DEBUG:boto:Port: 8000 2024-04-09T23:29:39.957 DEBUG:boto:Params: {} 2024-04-09T23:29:39.958 DEBUG:boto:Token: None 2024-04-09T23:29:39.958 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:29:39 GMT /xryqqk-63/ 2024-04-09T23:29:39.958 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DYU0O92Aji0V9r+Gnuc29lT4Ndk= 2024-04-09T23:29:39.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:29:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DYU0O92Aji0V9r+Gnuc29lT4Ndk=', 'Content-Length': '0'} 2024-04-09T23:29:39.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a634cc968ff6dead-006615cf63-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:29:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:29:39.961 DEBUG:boto:path=/xryqqk-63/dummy 2024-04-09T23:29:39.961 DEBUG:boto:auth_path=/xryqqk-63/dummy 2024-04-09T23:29:39.961 DEBUG:boto:Method: PUT 2024-04-09T23:29:39.961 DEBUG:boto:Path: /xryqqk-63/dummy 2024-04-09T23:29:39.961 DEBUG:boto:Data: 2024-04-09T23:29:39.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:29:39.961 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:29:39.961 DEBUG:boto:Port: 8000 2024-04-09T23:29:39.961 DEBUG:boto:Params: {} 2024-04-09T23:29:39.962 DEBUG:boto:Token: None 2024-04-09T23:29:39.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:29:39 GMT /xryqqk-63/dummy 2024-04-09T23:29:39.962 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:6GXtccg/BdFSnkczDc0x/oitOSk= 2024-04-09T23:29:39.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:29:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6GXtccg/BdFSnkczDc0x/oitOSk='} 2024-04-09T23:29:39.968 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000511b5e87734a0375-006615cf63-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:29:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:29:39.969 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-63 2024-04-09T23:29:39.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:29:39.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:29:39.969 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:29:40.026 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:29:40.026 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:29:40.066 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-63 2024-04-09T23:29:40.080 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-63 2024-04-09T23:29:40.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:29:40.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:29:40.081 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:29:40.138 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:29:40.138 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:29:40.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:40.188+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:29:45.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:45.188+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:29:50.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:50.189+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:29:55.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:29:55.190+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:00.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:00.190+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:05.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:05.191+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:10.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:10.192+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:15.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:15.192+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:20.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:20.193+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:25.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:25.193+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:30.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:30.194+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:35.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:35.195+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:40.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:40.194+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:45.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:45.195+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:50.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:50.196+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:30:55.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:30:55.196+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:00.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:00.197+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:05.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:05.197+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:10.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:10.198+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:15.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:15.198+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:20.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:20.199+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:25.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:25.199+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:30.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:30.200+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:35.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:35.200+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:40.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:40.201+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:45.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:45.202+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:50.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:50.202+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:31:55.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:31:55.203+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:00.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:00.203+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:05.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:05.205+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:10.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:10.205+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:15.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:15.204+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:20.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:20.205+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:25.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:25.206+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:30.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:30.206+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:35.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:35.207+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:40.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:40.207+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:45.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:45.208+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:50.208+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:32:55.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:32:55.209+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:00.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:00.209+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:05.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:05.210+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:10.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:10.211+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:15.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:15.211+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:20.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:20.212+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:25.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:25.213+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:30.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:30.214+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:35.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:35.214+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:40.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:40.215+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:45.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:45.216+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:50.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:50.216+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:33:55.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:33:55.216+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:00.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:00.217+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:05.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:05.217+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:10.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:10.218+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:15.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:15.218+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:20.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:20.219+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:25.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:25.219+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:30.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:30.220+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-09T23:34:35.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:35.220+0000 7f0f89895780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:34:35.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:35.220+0000 7f0f89895780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:34:35.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:35.220+0000 7f0f89895780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:34:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:34:35.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... FAIL 2024-04-09T23:34:35.250 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.251 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.251 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.251 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.251 DEBUG:boto:Data: 2024-04-09T23:34:35.251 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.251 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.251 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.251 DEBUG:boto:Params: {} 2024-04-09T23:34:35.252 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:34:35.252 DEBUG:boto:Token: None 2024-04-09T23:34:35.252 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.252 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tw63IuQ6HHRQv8MJGvQPeB7f1zQ= 2024-04-09T23:34:35.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tw63IuQ6HHRQv8MJGvQPeB7f1zQ=', 'Content-Length': '0'} 2024-04-09T23:34:35.286 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000014d252e525377c33-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.286 DEBUG:rgw_multi.tests:created bucket=xryqqk-64 2024-04-09T23:34:35.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:34:35.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:34:35.287 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:34:35.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:35.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:35.472 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:34:35.493 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705675.283043_1240.1", 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:34:35.283043Z" 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:34:35.495 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:34:35.496 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705675.283043_1240.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:34:35.496 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:34:35.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:34:35.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:34:35.497 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:34:35.547 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:35.548 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:35.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:35.590+0000 7fdfde81d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:34:35.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:35.591+0000 7fdfde81d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:34:35.609 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:34:35.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:34:35.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:34:35.612 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705675.283043_1240.1", 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:34:35.613 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:35.283043Z", 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:34:35.614 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:34:35.615 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:34:35Z" 2024-04-09T23:34:35.616 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:34:35.616 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705675.283043_1240.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:34:35.283043Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:34:35Z" } 2024-04-09T23:34:35.617 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705675.283043_1240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:34:35.283043Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:34:35.617 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705675.283043_1240.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:34:35.617 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705675.283043_1240.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:34:35.617 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:34:35.617 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.617 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.617 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.617 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.618 DEBUG:boto:Data: 2024-04-09T23:34:35.618 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.618 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.618 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.618 DEBUG:boto:Params: {} 2024-04-09T23:34:35.618 DEBUG:boto:Token: None 2024-04-09T23:34:35.618 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.618 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1ed2334079330d9-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.621 DEBUG:boto:path=/xryqqk-64/obj11 2024-04-09T23:34:35.621 DEBUG:boto:auth_path=/xryqqk-64/obj11 2024-04-09T23:34:35.621 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.621 DEBUG:boto:Path: /xryqqk-64/obj11 2024-04-09T23:34:35.621 DEBUG:boto:Data: 2024-04-09T23:34:35.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.622 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.622 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.622 DEBUG:boto:Params: {} 2024-04-09T23:34:35.622 DEBUG:boto:Token: None 2024-04-09T23:34:35.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj11 2024-04-09T23:34:35.622 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IDF6NWO/r0J4eHFSZwbpHqipkfU= 2024-04-09T23:34:35.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IDF6NWO/r0J4eHFSZwbpHqipkfU='} 2024-04-09T23:34:35.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c0356396e170990-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.635 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.635 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.635 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.635 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.635 DEBUG:boto:Data: 2024-04-09T23:34:35.635 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.635 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.635 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.635 DEBUG:boto:Params: {} 2024-04-09T23:34:35.635 DEBUG:boto:Token: None 2024-04-09T23:34:35.636 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.636 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3fd00b37e80d432-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.638 DEBUG:boto:path=/xryqqk-64/obj22 2024-04-09T23:34:35.638 DEBUG:boto:auth_path=/xryqqk-64/obj22 2024-04-09T23:34:35.638 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.639 DEBUG:boto:Path: /xryqqk-64/obj22 2024-04-09T23:34:35.639 DEBUG:boto:Data: 2024-04-09T23:34:35.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.639 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.639 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.639 DEBUG:boto:Params: {} 2024-04-09T23:34:35.639 DEBUG:boto:Token: None 2024-04-09T23:34:35.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj22 2024-04-09T23:34:35.639 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tdgQ5HpeStYowPptTl/1qbG5SAE= 2024-04-09T23:34:35.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tdgQ5HpeStYowPptTl/1qbG5SAE='} 2024-04-09T23:34:35.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067e05189d092a109-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.644 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.645 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.645 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.645 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.645 DEBUG:boto:Data: 2024-04-09T23:34:35.645 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.645 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.645 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.645 DEBUG:boto:Params: {} 2024-04-09T23:34:35.645 DEBUG:boto:Token: None 2024-04-09T23:34:35.645 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.646 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084c5f6fd2e9f2fdd-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.648 DEBUG:boto:path=/xryqqk-64/obj33 2024-04-09T23:34:35.648 DEBUG:boto:auth_path=/xryqqk-64/obj33 2024-04-09T23:34:35.648 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.648 DEBUG:boto:Path: /xryqqk-64/obj33 2024-04-09T23:34:35.648 DEBUG:boto:Data: 2024-04-09T23:34:35.649 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.649 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.649 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.649 DEBUG:boto:Params: {} 2024-04-09T23:34:35.649 DEBUG:boto:Token: None 2024-04-09T23:34:35.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj33 2024-04-09T23:34:35.649 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:a+rT6a916sLuj1QNad+HIDNN03o= 2024-04-09T23:34:35.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:a+rT6a916sLuj1QNad+HIDNN03o='} 2024-04-09T23:34:35.657 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000295a35bb9d932869-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.657 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.657 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.657 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.657 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.657 DEBUG:boto:Data: 2024-04-09T23:34:35.657 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.657 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.657 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.658 DEBUG:boto:Params: {} 2024-04-09T23:34:35.658 DEBUG:boto:Token: None 2024-04-09T23:34:35.658 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.658 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d129e3f865c7765e-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.661 DEBUG:boto:path=/xryqqk-64/obj44 2024-04-09T23:34:35.661 DEBUG:boto:auth_path=/xryqqk-64/obj44 2024-04-09T23:34:35.661 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.661 DEBUG:boto:Path: /xryqqk-64/obj44 2024-04-09T23:34:35.661 DEBUG:boto:Data: 2024-04-09T23:34:35.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.661 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.661 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.661 DEBUG:boto:Params: {} 2024-04-09T23:34:35.661 DEBUG:boto:Token: None 2024-04-09T23:34:35.661 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj44 2024-04-09T23:34:35.662 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:k79K67CQlXcvC60iXFU4WO42wRI= 2024-04-09T23:34:35.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k79K67CQlXcvC60iXFU4WO42wRI='} 2024-04-09T23:34:35.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bfb615b997c44e3-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.707 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.707 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.707 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.707 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.707 DEBUG:boto:Data: 2024-04-09T23:34:35.707 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.707 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.707 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.708 DEBUG:boto:Params: {} 2024-04-09T23:34:35.708 DEBUG:boto:Token: None 2024-04-09T23:34:35.708 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.708 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000739c58cceca45116-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.710 DEBUG:boto:path=/xryqqk-64/obj55 2024-04-09T23:34:35.710 DEBUG:boto:auth_path=/xryqqk-64/obj55 2024-04-09T23:34:35.710 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.711 DEBUG:boto:Path: /xryqqk-64/obj55 2024-04-09T23:34:35.711 DEBUG:boto:Data: 2024-04-09T23:34:35.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.711 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.711 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.711 DEBUG:boto:Params: {} 2024-04-09T23:34:35.711 DEBUG:boto:Token: None 2024-04-09T23:34:35.711 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj55 2024-04-09T23:34:35.711 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:chYURlqhjraLWrGmPNOZbJQIypY= 2024-04-09T23:34:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:chYURlqhjraLWrGmPNOZbJQIypY='} 2024-04-09T23:34:35.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e9ee2c3a7f839b7-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:35.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:35.716 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:34:35.771 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:35.772 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:35.845 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:34:35.846 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-64 2024-04-09T23:34:35.849 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 5 2024-04-09T23:34:35.907 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.907 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.908 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.908 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.908 DEBUG:boto:Data: 2024-04-09T23:34:35.908 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.908 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.908 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.908 DEBUG:boto:Params: {} 2024-04-09T23:34:35.908 DEBUG:boto:Token: None 2024-04-09T23:34:35.909 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.909 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095bff9d38657d6c8-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.912 DEBUG:boto:path=/xryqqk-64/obj17 2024-04-09T23:34:35.912 DEBUG:boto:auth_path=/xryqqk-64/obj17 2024-04-09T23:34:35.912 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.912 DEBUG:boto:Path: /xryqqk-64/obj17 2024-04-09T23:34:35.912 DEBUG:boto:Data: 2024-04-09T23:34:35.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.912 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.912 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.913 DEBUG:boto:Params: {} 2024-04-09T23:34:35.913 DEBUG:boto:Token: None 2024-04-09T23:34:35.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj17 2024-04-09T23:34:35.913 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ziig3LRYwUO0wMiDk1AA5EEeDLE= 2024-04-09T23:34:35.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ziig3LRYwUO0wMiDk1AA5EEeDLE='} 2024-04-09T23:34:35.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5705aa950e166bc-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.921 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.921 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.921 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.922 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.922 DEBUG:boto:Data: 2024-04-09T23:34:35.922 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.922 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.922 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.922 DEBUG:boto:Params: {} 2024-04-09T23:34:35.922 DEBUG:boto:Token: None 2024-04-09T23:34:35.922 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.923 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.925 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001383461bd0d2edc2-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.925 DEBUG:boto:path=/xryqqk-64/obj34 2024-04-09T23:34:35.926 DEBUG:boto:auth_path=/xryqqk-64/obj34 2024-04-09T23:34:35.926 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.926 DEBUG:boto:Path: /xryqqk-64/obj34 2024-04-09T23:34:35.926 DEBUG:boto:Data: 2024-04-09T23:34:35.926 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.926 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.926 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.926 DEBUG:boto:Params: {} 2024-04-09T23:34:35.927 DEBUG:boto:Token: None 2024-04-09T23:34:35.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj34 2024-04-09T23:34:35.927 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nCPcF899NL1dHcs/Wrzh4sfSrqY= 2024-04-09T23:34:35.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nCPcF899NL1dHcs/Wrzh4sfSrqY='} 2024-04-09T23:34:35.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003536fc4184fb2950-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.971 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.971 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.971 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.971 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.971 DEBUG:boto:Data: 2024-04-09T23:34:35.971 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.971 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.972 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.972 DEBUG:boto:Params: {} 2024-04-09T23:34:35.972 DEBUG:boto:Token: None 2024-04-09T23:34:35.972 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.972 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070ac447d38d97ef2-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.975 DEBUG:boto:path=/xryqqk-64/obj51 2024-04-09T23:34:35.975 DEBUG:boto:auth_path=/xryqqk-64/obj51 2024-04-09T23:34:35.975 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.975 DEBUG:boto:Path: /xryqqk-64/obj51 2024-04-09T23:34:35.975 DEBUG:boto:Data: 2024-04-09T23:34:35.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.975 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.976 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.976 DEBUG:boto:Params: {} 2024-04-09T23:34:35.976 DEBUG:boto:Token: None 2024-04-09T23:34:35.976 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj51 2024-04-09T23:34:35.976 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1zDB7ToT0ZqdfAFPnXhs8edGBf4= 2024-04-09T23:34:35.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1zDB7ToT0ZqdfAFPnXhs8edGBf4='} 2024-04-09T23:34:35.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9837026c264d9b1-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.987 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:35.987 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:35.988 DEBUG:boto:Method: HEAD 2024-04-09T23:34:35.988 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:35.988 DEBUG:boto:Data: 2024-04-09T23:34:35.988 DEBUG:boto:Headers: {} 2024-04-09T23:34:35.988 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.988 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.988 DEBUG:boto:Params: {} 2024-04-09T23:34:35.989 DEBUG:boto:Token: None 2024-04-09T23:34:35.989 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/ 2024-04-09T23:34:35.989 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-09T23:34:35.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-09T23:34:35.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7d0a83ae885d11-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:35.992 DEBUG:boto:path=/xryqqk-64/obj68 2024-04-09T23:34:35.992 DEBUG:boto:auth_path=/xryqqk-64/obj68 2024-04-09T23:34:35.992 DEBUG:boto:Method: PUT 2024-04-09T23:34:35.992 DEBUG:boto:Path: /xryqqk-64/obj68 2024-04-09T23:34:35.992 DEBUG:boto:Data: 2024-04-09T23:34:35.992 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:35.992 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:35.992 DEBUG:boto:Port: 8000 2024-04-09T23:34:35.992 DEBUG:boto:Params: {} 2024-04-09T23:34:35.993 DEBUG:boto:Token: None 2024-04-09T23:34:35.993 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:35 GMT /xryqqk-64/obj68 2024-04-09T23:34:35.993 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PpA9eB/3nydt93EKfo+/4UnyRBs= 2024-04-09T23:34:35.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PpA9eB/3nydt93EKfo+/4UnyRBs='} 2024-04-09T23:34:36.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f03931dfb51191a5-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.039 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.039 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.039 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.039 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.039 DEBUG:boto:Data: 2024-04-09T23:34:36.039 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.039 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.040 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.040 DEBUG:boto:Params: {} 2024-04-09T23:34:36.040 DEBUG:boto:Token: None 2024-04-09T23:34:36.040 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.040 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4d7a73640ea5201-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.043 DEBUG:boto:path=/xryqqk-64/obj85 2024-04-09T23:34:36.043 DEBUG:boto:auth_path=/xryqqk-64/obj85 2024-04-09T23:34:36.043 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.043 DEBUG:boto:Path: /xryqqk-64/obj85 2024-04-09T23:34:36.043 DEBUG:boto:Data: 2024-04-09T23:34:36.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.044 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.044 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.044 DEBUG:boto:Params: {} 2024-04-09T23:34:36.044 DEBUG:boto:Token: None 2024-04-09T23:34:36.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj85 2024-04-09T23:34:36.044 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Z76g5zGzd6q2TUppdfZxBakEEg8= 2024-04-09T23:34:36.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Z76g5zGzd6q2TUppdfZxBakEEg8='} 2024-04-09T23:34:36.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054836dd1af3191fd-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:36.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:36.050 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:34:36.106 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:36.106 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:36.168 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:34:36.168 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-64 2024-04-09T23:34:36.173 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 10 2024-04-09T23:34:36.236 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.236 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.236 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.237 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.237 DEBUG:boto:Data: 2024-04-09T23:34:36.237 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.237 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.237 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.237 DEBUG:boto:Params: {} 2024-04-09T23:34:36.237 DEBUG:boto:Token: None 2024-04-09T23:34:36.237 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.238 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.241 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5cca92d07be59cc-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.241 DEBUG:boto:path=/xryqqk-64/obj19 2024-04-09T23:34:36.241 DEBUG:boto:auth_path=/xryqqk-64/obj19 2024-04-09T23:34:36.241 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.241 DEBUG:boto:Path: /xryqqk-64/obj19 2024-04-09T23:34:36.241 DEBUG:boto:Data: 2024-04-09T23:34:36.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.242 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.242 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.242 DEBUG:boto:Params: {} 2024-04-09T23:34:36.242 DEBUG:boto:Token: None 2024-04-09T23:34:36.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj19 2024-04-09T23:34:36.242 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YnC9zango27PPT2+3CPCORs2pmo= 2024-04-09T23:34:36.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YnC9zango27PPT2+3CPCORs2pmo='} 2024-04-09T23:34:36.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a170b77f5886728-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.248 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.248 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.248 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.249 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.249 DEBUG:boto:Data: 2024-04-09T23:34:36.249 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.249 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.249 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.249 DEBUG:boto:Params: {} 2024-04-09T23:34:36.249 DEBUG:boto:Token: None 2024-04-09T23:34:36.249 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.249 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000305bbb07eb39dd36-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.252 DEBUG:boto:path=/xryqqk-64/obj38 2024-04-09T23:34:36.252 DEBUG:boto:auth_path=/xryqqk-64/obj38 2024-04-09T23:34:36.253 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.253 DEBUG:boto:Path: /xryqqk-64/obj38 2024-04-09T23:34:36.253 DEBUG:boto:Data: 2024-04-09T23:34:36.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.253 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.253 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.253 DEBUG:boto:Params: {} 2024-04-09T23:34:36.253 DEBUG:boto:Token: None 2024-04-09T23:34:36.253 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj38 2024-04-09T23:34:36.254 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:2lakgoHynB2dKdwpivhlDhh3PO0= 2024-04-09T23:34:36.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2lakgoHynB2dKdwpivhlDhh3PO0='} 2024-04-09T23:34:36.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45b48e618b42ecb-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.299 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.299 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.299 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.299 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.299 DEBUG:boto:Data: 2024-04-09T23:34:36.299 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.299 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.299 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.300 DEBUG:boto:Params: {} 2024-04-09T23:34:36.300 DEBUG:boto:Token: None 2024-04-09T23:34:36.300 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.300 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ea95934059a07c7-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.303 DEBUG:boto:path=/xryqqk-64/obj57 2024-04-09T23:34:36.304 DEBUG:boto:auth_path=/xryqqk-64/obj57 2024-04-09T23:34:36.304 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.304 DEBUG:boto:Path: /xryqqk-64/obj57 2024-04-09T23:34:36.304 DEBUG:boto:Data: 2024-04-09T23:34:36.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.304 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.304 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.304 DEBUG:boto:Params: {} 2024-04-09T23:34:36.304 DEBUG:boto:Token: None 2024-04-09T23:34:36.305 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj57 2024-04-09T23:34:36.305 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xZbjDFvCPvr3ebraKS/ggAoE+N0= 2024-04-09T23:34:36.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xZbjDFvCPvr3ebraKS/ggAoE+N0='} 2024-04-09T23:34:36.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000defaf95de22aa461-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.309 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.309 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.309 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.309 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.309 DEBUG:boto:Data: 2024-04-09T23:34:36.309 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.309 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.309 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.309 DEBUG:boto:Params: {} 2024-04-09T23:34:36.310 DEBUG:boto:Token: None 2024-04-09T23:34:36.310 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.310 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dd7f4974297b18f-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.313 DEBUG:boto:path=/xryqqk-64/obj76 2024-04-09T23:34:36.313 DEBUG:boto:auth_path=/xryqqk-64/obj76 2024-04-09T23:34:36.313 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.313 DEBUG:boto:Path: /xryqqk-64/obj76 2024-04-09T23:34:36.313 DEBUG:boto:Data: 2024-04-09T23:34:36.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.313 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.314 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.314 DEBUG:boto:Params: {} 2024-04-09T23:34:36.314 DEBUG:boto:Token: None 2024-04-09T23:34:36.314 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj76 2024-04-09T23:34:36.314 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:pMc3lO/Evzdvb95uctGRlcjBKPg= 2024-04-09T23:34:36.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pMc3lO/Evzdvb95uctGRlcjBKPg='} 2024-04-09T23:34:36.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079b1af0b99f1828a-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.359 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.359 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.359 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.359 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.359 DEBUG:boto:Data: 2024-04-09T23:34:36.359 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.359 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.359 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.359 DEBUG:boto:Params: {} 2024-04-09T23:34:36.360 DEBUG:boto:Token: None 2024-04-09T23:34:36.360 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.360 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.363 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83a1280d844dafb-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.363 DEBUG:boto:path=/xryqqk-64/obj95 2024-04-09T23:34:36.363 DEBUG:boto:auth_path=/xryqqk-64/obj95 2024-04-09T23:34:36.363 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.364 DEBUG:boto:Path: /xryqqk-64/obj95 2024-04-09T23:34:36.364 DEBUG:boto:Data: 2024-04-09T23:34:36.364 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.364 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.364 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.364 DEBUG:boto:Params: {} 2024-04-09T23:34:36.364 DEBUG:boto:Token: None 2024-04-09T23:34:36.364 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj95 2024-04-09T23:34:36.365 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:hfwP9/EYmNFhmbRzGMXN+lMj63M= 2024-04-09T23:34:36.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hfwP9/EYmNFhmbRzGMXN+lMj63M='} 2024-04-09T23:34:36.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8ce039c5769f215-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:36.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:36.369 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:34:36.422 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:36.422 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:36.486 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:34:36.486 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-64 2024-04-09T23:34:36.493 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 15 2024-04-09T23:34:36.588 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.588 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.588 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.588 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.588 DEBUG:boto:Data: 2024-04-09T23:34:36.588 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.588 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.588 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.589 DEBUG:boto:Params: {} 2024-04-09T23:34:36.589 DEBUG:boto:Token: None 2024-04-09T23:34:36.589 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.589 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.592 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001159cc04f75bdbe4-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.593 DEBUG:boto:path=/xryqqk-64/obj23 2024-04-09T23:34:36.593 DEBUG:boto:auth_path=/xryqqk-64/obj23 2024-04-09T23:34:36.593 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.593 DEBUG:boto:Path: /xryqqk-64/obj23 2024-04-09T23:34:36.593 DEBUG:boto:Data: 2024-04-09T23:34:36.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.593 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.593 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.594 DEBUG:boto:Params: {} 2024-04-09T23:34:36.594 DEBUG:boto:Token: None 2024-04-09T23:34:36.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj23 2024-04-09T23:34:36.594 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:u2e1iwTtA25czYHn1iUTOdwEA+E= 2024-04-09T23:34:36.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u2e1iwTtA25czYHn1iUTOdwEA+E='} 2024-04-09T23:34:36.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b68577430e9e8ac6-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.643 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.643 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.643 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.643 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.643 DEBUG:boto:Data: 2024-04-09T23:34:36.643 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.643 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.643 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.643 DEBUG:boto:Params: {} 2024-04-09T23:34:36.644 DEBUG:boto:Token: None 2024-04-09T23:34:36.644 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.644 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039670f7b54b76a60-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.648 DEBUG:boto:path=/xryqqk-64/obj46 2024-04-09T23:34:36.648 DEBUG:boto:auth_path=/xryqqk-64/obj46 2024-04-09T23:34:36.648 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.648 DEBUG:boto:Path: /xryqqk-64/obj46 2024-04-09T23:34:36.648 DEBUG:boto:Data: 2024-04-09T23:34:36.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.649 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.649 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.649 DEBUG:boto:Params: {} 2024-04-09T23:34:36.649 DEBUG:boto:Token: None 2024-04-09T23:34:36.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj46 2024-04-09T23:34:36.649 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/ojwIYoaE/QX/w0WZmXBo9VqTig= 2024-04-09T23:34:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ojwIYoaE/QX/w0WZmXBo9VqTig='} 2024-04-09T23:34:36.655 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000776094936df42fd4-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.655 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.655 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.655 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.656 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.656 DEBUG:boto:Data: 2024-04-09T23:34:36.656 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.656 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.656 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.656 DEBUG:boto:Params: {} 2024-04-09T23:34:36.656 DEBUG:boto:Token: None 2024-04-09T23:34:36.656 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.656 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be1cab099fe8f02-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.659 DEBUG:boto:path=/xryqqk-64/obj69 2024-04-09T23:34:36.659 DEBUG:boto:auth_path=/xryqqk-64/obj69 2024-04-09T23:34:36.659 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.660 DEBUG:boto:Path: /xryqqk-64/obj69 2024-04-09T23:34:36.660 DEBUG:boto:Data: 2024-04-09T23:34:36.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.660 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.660 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.660 DEBUG:boto:Params: {} 2024-04-09T23:34:36.660 DEBUG:boto:Token: None 2024-04-09T23:34:36.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj69 2024-04-09T23:34:36.660 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SYrNP2sIsBLI8HAX/5I/PrxhUMc= 2024-04-09T23:34:36.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SYrNP2sIsBLI8HAX/5I/PrxhUMc='} 2024-04-09T23:34:36.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2affa1733dbdab7-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.707 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.707 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.707 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.707 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.708 DEBUG:boto:Data: 2024-04-09T23:34:36.708 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.708 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.708 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.708 DEBUG:boto:Params: {} 2024-04-09T23:34:36.708 DEBUG:boto:Token: None 2024-04-09T23:34:36.708 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.708 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001df36eab3a002072-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.712 DEBUG:boto:path=/xryqqk-64/obj92 2024-04-09T23:34:36.712 DEBUG:boto:auth_path=/xryqqk-64/obj92 2024-04-09T23:34:36.712 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.712 DEBUG:boto:Path: /xryqqk-64/obj92 2024-04-09T23:34:36.712 DEBUG:boto:Data: 2024-04-09T23:34:36.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.712 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.712 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.712 DEBUG:boto:Params: {} 2024-04-09T23:34:36.712 DEBUG:boto:Token: None 2024-04-09T23:34:36.713 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj92 2024-04-09T23:34:36.713 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:x1zMmTVxt1tkixHdAJ5JP0Hr/pw= 2024-04-09T23:34:36.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x1zMmTVxt1tkixHdAJ5JP0Hr/pw='} 2024-04-09T23:34:36.719 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092bf78b7e6216f41-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.719 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.719 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.719 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.719 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.719 DEBUG:boto:Data: 2024-04-09T23:34:36.719 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.719 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.719 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.719 DEBUG:boto:Params: {} 2024-04-09T23:34:36.720 DEBUG:boto:Token: None 2024-04-09T23:34:36.720 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.720 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004df3d97750c65999-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.723 DEBUG:boto:path=/xryqqk-64/obj115 2024-04-09T23:34:36.723 DEBUG:boto:auth_path=/xryqqk-64/obj115 2024-04-09T23:34:36.723 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.723 DEBUG:boto:Path: /xryqqk-64/obj115 2024-04-09T23:34:36.723 DEBUG:boto:Data: 2024-04-09T23:34:36.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.724 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.724 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.724 DEBUG:boto:Params: {} 2024-04-09T23:34:36.724 DEBUG:boto:Token: None 2024-04-09T23:34:36.724 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj115 2024-04-09T23:34:36.725 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RmSmU68XCYMK3LoYsUopW+KAyTg= 2024-04-09T23:34:36.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RmSmU68XCYMK3LoYsUopW+KAyTg='} 2024-04-09T23:34:36.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee8359bda8e7b010-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:36.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:36.767 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:34:36.821 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:36.822 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:36.882 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:34:36.883 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-64 2024-04-09T23:34:36.891 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 20 2024-04-09T23:34:36.979 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.980 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.980 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.980 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.980 DEBUG:boto:Data: 2024-04-09T23:34:36.980 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.980 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.980 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.980 DEBUG:boto:Params: {} 2024-04-09T23:34:36.980 DEBUG:boto:Token: None 2024-04-09T23:34:36.981 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.981 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.984 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa2b801e9326770f-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.984 DEBUG:boto:path=/xryqqk-64/obj29 2024-04-09T23:34:36.984 DEBUG:boto:auth_path=/xryqqk-64/obj29 2024-04-09T23:34:36.984 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.984 DEBUG:boto:Path: /xryqqk-64/obj29 2024-04-09T23:34:36.985 DEBUG:boto:Data: 2024-04-09T23:34:36.985 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.985 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.985 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.985 DEBUG:boto:Params: {} 2024-04-09T23:34:36.985 DEBUG:boto:Token: None 2024-04-09T23:34:36.985 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj29 2024-04-09T23:34:36.985 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:juepciHOyuiEYlYu655PMOtksvk= 2024-04-09T23:34:36.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:juepciHOyuiEYlYu655PMOtksvk='} 2024-04-09T23:34:36.992 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050734d95420946d5-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.992 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:36.992 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:36.992 DEBUG:boto:Method: HEAD 2024-04-09T23:34:36.992 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:36.992 DEBUG:boto:Data: 2024-04-09T23:34:36.992 DEBUG:boto:Headers: {} 2024-04-09T23:34:36.993 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.993 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.993 DEBUG:boto:Params: {} 2024-04-09T23:34:36.993 DEBUG:boto:Token: None 2024-04-09T23:34:36.993 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/ 2024-04-09T23:34:36.993 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-09T23:34:36.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-09T23:34:36.996 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066d007adac11b25a-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:36.996 DEBUG:boto:path=/xryqqk-64/obj58 2024-04-09T23:34:36.997 DEBUG:boto:auth_path=/xryqqk-64/obj58 2024-04-09T23:34:36.997 DEBUG:boto:Method: PUT 2024-04-09T23:34:36.997 DEBUG:boto:Path: /xryqqk-64/obj58 2024-04-09T23:34:36.997 DEBUG:boto:Data: 2024-04-09T23:34:36.997 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:36.997 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:36.997 DEBUG:boto:Port: 8000 2024-04-09T23:34:36.997 DEBUG:boto:Params: {} 2024-04-09T23:34:36.997 DEBUG:boto:Token: None 2024-04-09T23:34:36.998 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:36 GMT /xryqqk-64/obj58 2024-04-09T23:34:36.998 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qWpjETrvg+wa6hkCq5Cj0oYE6L4= 2024-04-09T23:34:36.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qWpjETrvg+wa6hkCq5Cj0oYE6L4='} 2024-04-09T23:34:37.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8d851bb805d4d14-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.043 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.043 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.043 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.043 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.044 DEBUG:boto:Data: 2024-04-09T23:34:37.044 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.044 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.044 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.044 DEBUG:boto:Params: {} 2024-04-09T23:34:37.045 DEBUG:boto:Token: None 2024-04-09T23:34:37.045 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.045 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.048 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba3020747287cb58-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.048 DEBUG:boto:path=/xryqqk-64/obj87 2024-04-09T23:34:37.048 DEBUG:boto:auth_path=/xryqqk-64/obj87 2024-04-09T23:34:37.049 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.049 DEBUG:boto:Path: /xryqqk-64/obj87 2024-04-09T23:34:37.049 DEBUG:boto:Data: 2024-04-09T23:34:37.049 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.049 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.049 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.049 DEBUG:boto:Params: {} 2024-04-09T23:34:37.049 DEBUG:boto:Token: None 2024-04-09T23:34:37.049 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj87 2024-04-09T23:34:37.050 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:J3E3P+rBCWhbY9oqU+S5Hb1rL6I= 2024-04-09T23:34:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J3E3P+rBCWhbY9oqU+S5Hb1rL6I='} 2024-04-09T23:34:37.056 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6c26aaed0ce32d5-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.056 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.056 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.056 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.056 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.056 DEBUG:boto:Data: 2024-04-09T23:34:37.056 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.057 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.057 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.057 DEBUG:boto:Params: {} 2024-04-09T23:34:37.057 DEBUG:boto:Token: None 2024-04-09T23:34:37.057 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.058 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f57595421f61b8fd-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.061 DEBUG:boto:path=/xryqqk-64/obj116 2024-04-09T23:34:37.062 DEBUG:boto:auth_path=/xryqqk-64/obj116 2024-04-09T23:34:37.062 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.062 DEBUG:boto:Path: /xryqqk-64/obj116 2024-04-09T23:34:37.062 DEBUG:boto:Data: 2024-04-09T23:34:37.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.062 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.062 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.062 DEBUG:boto:Params: {} 2024-04-09T23:34:37.063 DEBUG:boto:Token: None 2024-04-09T23:34:37.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj116 2024-04-09T23:34:37.063 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mRsVH0p++4dZbopMUeXvCV6z0QE= 2024-04-09T23:34:37.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mRsVH0p++4dZbopMUeXvCV6z0QE='} 2024-04-09T23:34:37.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001942a65b68654482-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.107 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.107 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.107 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.107 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.107 DEBUG:boto:Data: 2024-04-09T23:34:37.107 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.107 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.107 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.107 DEBUG:boto:Params: {} 2024-04-09T23:34:37.107 DEBUG:boto:Token: None 2024-04-09T23:34:37.108 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.108 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000556850af91727f6e-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.111 DEBUG:boto:path=/xryqqk-64/obj145 2024-04-09T23:34:37.111 DEBUG:boto:auth_path=/xryqqk-64/obj145 2024-04-09T23:34:37.111 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.111 DEBUG:boto:Path: /xryqqk-64/obj145 2024-04-09T23:34:37.111 DEBUG:boto:Data: 2024-04-09T23:34:37.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.111 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.111 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.111 DEBUG:boto:Params: {} 2024-04-09T23:34:37.111 DEBUG:boto:Token: None 2024-04-09T23:34:37.112 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj145 2024-04-09T23:34:37.112 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rjC+CqK0PMgoJ9s3zncg9IBQ3io= 2024-04-09T23:34:37.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rjC+CqK0PMgoJ9s3zncg9IBQ3io='} 2024-04-09T23:34:37.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dab8de6c534541b4-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:37.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:37.119 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:34:37.173 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:37.173 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:37.241 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:34:37.241 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-64 2024-04-09T23:34:37.252 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 25 2024-04-09T23:34:37.342 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.342 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.342 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.342 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.342 DEBUG:boto:Data: 2024-04-09T23:34:37.342 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.342 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.342 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.343 DEBUG:boto:Params: {} 2024-04-09T23:34:37.343 DEBUG:boto:Token: None 2024-04-09T23:34:37.343 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.343 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cca077a743a9a8f-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.347 DEBUG:boto:path=/xryqqk-64/obj31 2024-04-09T23:34:37.347 DEBUG:boto:auth_path=/xryqqk-64/obj31 2024-04-09T23:34:37.347 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.347 DEBUG:boto:Path: /xryqqk-64/obj31 2024-04-09T23:34:37.347 DEBUG:boto:Data: 2024-04-09T23:34:37.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.348 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.348 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.348 DEBUG:boto:Params: {} 2024-04-09T23:34:37.348 DEBUG:boto:Token: None 2024-04-09T23:34:37.348 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj31 2024-04-09T23:34:37.348 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rpEbOPi/T2U2f/ipgOFpGCcwHJo= 2024-04-09T23:34:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rpEbOPi/T2U2f/ipgOFpGCcwHJo='} 2024-04-09T23:34:37.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7122bdbf7b5aeb0-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.391 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.391 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.391 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.391 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.391 DEBUG:boto:Data: 2024-04-09T23:34:37.391 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.391 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.391 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.391 DEBUG:boto:Params: {} 2024-04-09T23:34:37.391 DEBUG:boto:Token: None 2024-04-09T23:34:37.392 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.392 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.395 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008eece30e08ac2a31-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.395 DEBUG:boto:path=/xryqqk-64/obj62 2024-04-09T23:34:37.395 DEBUG:boto:auth_path=/xryqqk-64/obj62 2024-04-09T23:34:37.395 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.395 DEBUG:boto:Path: /xryqqk-64/obj62 2024-04-09T23:34:37.395 DEBUG:boto:Data: 2024-04-09T23:34:37.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.396 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.396 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.396 DEBUG:boto:Params: {} 2024-04-09T23:34:37.396 DEBUG:boto:Token: None 2024-04-09T23:34:37.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj62 2024-04-09T23:34:37.396 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:z5aoDK3CMtp+vZ49HaRCbYSzd2Y= 2024-04-09T23:34:37.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:z5aoDK3CMtp+vZ49HaRCbYSzd2Y='} 2024-04-09T23:34:37.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001ff8f405a2d6c32-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.403 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.403 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.403 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.403 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.403 DEBUG:boto:Data: 2024-04-09T23:34:37.403 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.403 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.403 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.403 DEBUG:boto:Params: {} 2024-04-09T23:34:37.403 DEBUG:boto:Token: None 2024-04-09T23:34:37.403 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.404 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a39db7690075fe4-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.407 DEBUG:boto:path=/xryqqk-64/obj93 2024-04-09T23:34:37.407 DEBUG:boto:auth_path=/xryqqk-64/obj93 2024-04-09T23:34:37.407 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.407 DEBUG:boto:Path: /xryqqk-64/obj93 2024-04-09T23:34:37.407 DEBUG:boto:Data: 2024-04-09T23:34:37.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.408 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.408 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.408 DEBUG:boto:Params: {} 2024-04-09T23:34:37.408 DEBUG:boto:Token: None 2024-04-09T23:34:37.408 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj93 2024-04-09T23:34:37.408 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:atFlCq+jB6FzDtW+fT6xjE0wx1Y= 2024-04-09T23:34:37.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:atFlCq+jB6FzDtW+fT6xjE0wx1Y='} 2024-04-09T23:34:37.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057bc48e535bbe36e-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.450 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.451 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.451 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.451 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.451 DEBUG:boto:Data: 2024-04-09T23:34:37.451 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.451 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.451 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.451 DEBUG:boto:Params: {} 2024-04-09T23:34:37.451 DEBUG:boto:Token: None 2024-04-09T23:34:37.451 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.451 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d885ffc928bff24-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.455 DEBUG:boto:path=/xryqqk-64/obj124 2024-04-09T23:34:37.455 DEBUG:boto:auth_path=/xryqqk-64/obj124 2024-04-09T23:34:37.455 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.455 DEBUG:boto:Path: /xryqqk-64/obj124 2024-04-09T23:34:37.455 DEBUG:boto:Data: 2024-04-09T23:34:37.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.455 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.455 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.455 DEBUG:boto:Params: {} 2024-04-09T23:34:37.455 DEBUG:boto:Token: None 2024-04-09T23:34:37.456 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj124 2024-04-09T23:34:37.456 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3MosNn5w7k0SpykX94ARWepPzoQ= 2024-04-09T23:34:37.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3MosNn5w7k0SpykX94ARWepPzoQ='} 2024-04-09T23:34:37.461 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e29d28d5ed56c8cb-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.461 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.461 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.461 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.461 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.461 DEBUG:boto:Data: 2024-04-09T23:34:37.461 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.462 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.462 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.462 DEBUG:boto:Params: {} 2024-04-09T23:34:37.462 DEBUG:boto:Token: None 2024-04-09T23:34:37.462 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.462 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9f4086980dc2d1b-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.466 DEBUG:boto:path=/xryqqk-64/obj155 2024-04-09T23:34:37.466 DEBUG:boto:auth_path=/xryqqk-64/obj155 2024-04-09T23:34:37.466 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.466 DEBUG:boto:Path: /xryqqk-64/obj155 2024-04-09T23:34:37.466 DEBUG:boto:Data: 2024-04-09T23:34:37.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.466 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.466 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.466 DEBUG:boto:Params: {} 2024-04-09T23:34:37.467 DEBUG:boto:Token: None 2024-04-09T23:34:37.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj155 2024-04-09T23:34:37.467 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:XlUTGErfodhlvEHObyyh+ZEIrMo= 2024-04-09T23:34:37.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XlUTGErfodhlvEHObyyh+ZEIrMo='} 2024-04-09T23:34:37.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b3d2b00ecb0b159-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:37.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:34:37.511 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:34:37.563 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:37.563 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:37.629 INFO:teuthology.orchestra.run.smithi063.stdout:tenant: 2024-04-09T23:34:37.629 INFO:teuthology.orchestra.run.smithi063.stdout:bucket name: xryqqk-64 2024-04-09T23:34:37.641 INFO:teuthology.orchestra.run.smithi063.stdout:total entries: 30 2024-04-09T23:34:37.740 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.740 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.740 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.740 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.740 DEBUG:boto:Data: 2024-04-09T23:34:37.740 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.740 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.740 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.740 DEBUG:boto:Params: {} 2024-04-09T23:34:37.741 DEBUG:boto:Token: None 2024-04-09T23:34:37.741 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.741 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e498c88a42cc5281-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.745 DEBUG:boto:path=/xryqqk-64/obj37 2024-04-09T23:34:37.745 DEBUG:boto:auth_path=/xryqqk-64/obj37 2024-04-09T23:34:37.745 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.745 DEBUG:boto:Path: /xryqqk-64/obj37 2024-04-09T23:34:37.745 DEBUG:boto:Data: 2024-04-09T23:34:37.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.745 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.745 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.745 DEBUG:boto:Params: {} 2024-04-09T23:34:37.746 DEBUG:boto:Token: None 2024-04-09T23:34:37.746 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj37 2024-04-09T23:34:37.746 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b/7WV2812o39jBioYbdD7YvR89c= 2024-04-09T23:34:37.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b/7WV2812o39jBioYbdD7YvR89c='} 2024-04-09T23:34:37.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000120ec679c7d62170-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.752 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.752 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.752 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.752 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.752 DEBUG:boto:Data: 2024-04-09T23:34:37.752 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.752 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.752 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.752 DEBUG:boto:Params: {} 2024-04-09T23:34:37.753 DEBUG:boto:Token: None 2024-04-09T23:34:37.753 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.753 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.757 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ff57aaa1abb79b0-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.757 DEBUG:boto:path=/xryqqk-64/obj74 2024-04-09T23:34:37.757 DEBUG:boto:auth_path=/xryqqk-64/obj74 2024-04-09T23:34:37.757 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.757 DEBUG:boto:Path: /xryqqk-64/obj74 2024-04-09T23:34:37.757 DEBUG:boto:Data: 2024-04-09T23:34:37.758 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.758 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.758 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.758 DEBUG:boto:Params: {} 2024-04-09T23:34:37.758 DEBUG:boto:Token: None 2024-04-09T23:34:37.758 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj74 2024-04-09T23:34:37.758 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:GRngGtyVS4HdzcXzYadezOje2SI= 2024-04-09T23:34:37.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GRngGtyVS4HdzcXzYadezOje2SI='} 2024-04-09T23:34:37.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026ec3c6b2334e326-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.803 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.803 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.803 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.803 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.803 DEBUG:boto:Data: 2024-04-09T23:34:37.803 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.804 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.804 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.804 DEBUG:boto:Params: {} 2024-04-09T23:34:37.804 DEBUG:boto:Token: None 2024-04-09T23:34:37.804 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.804 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9786b171b9e74ab-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.808 DEBUG:boto:path=/xryqqk-64/obj111 2024-04-09T23:34:37.808 DEBUG:boto:auth_path=/xryqqk-64/obj111 2024-04-09T23:34:37.808 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.808 DEBUG:boto:Path: /xryqqk-64/obj111 2024-04-09T23:34:37.809 DEBUG:boto:Data: 2024-04-09T23:34:37.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.809 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.809 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.809 DEBUG:boto:Params: {} 2024-04-09T23:34:37.809 DEBUG:boto:Token: None 2024-04-09T23:34:37.809 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj111 2024-04-09T23:34:37.809 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:f/cEKaRlxFEcmeHMWVJg/MRqW8o= 2024-04-09T23:34:37.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:f/cEKaRlxFEcmeHMWVJg/MRqW8o='} 2024-04-09T23:34:37.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe06410da11ccebf-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.815 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.815 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.815 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.815 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.815 DEBUG:boto:Data: 2024-04-09T23:34:37.815 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.815 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.815 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.815 DEBUG:boto:Params: {} 2024-04-09T23:34:37.815 DEBUG:boto:Token: None 2024-04-09T23:34:37.816 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.816 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000363fff476e221c1c-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.819 DEBUG:boto:path=/xryqqk-64/obj148 2024-04-09T23:34:37.820 DEBUG:boto:auth_path=/xryqqk-64/obj148 2024-04-09T23:34:37.820 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.820 DEBUG:boto:Path: /xryqqk-64/obj148 2024-04-09T23:34:37.820 DEBUG:boto:Data: 2024-04-09T23:34:37.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.820 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.820 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.820 DEBUG:boto:Params: {} 2024-04-09T23:34:37.820 DEBUG:boto:Token: None 2024-04-09T23:34:37.820 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj148 2024-04-09T23:34:37.821 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zysad6eQ3PIARtveyPfSa787Sgc= 2024-04-09T23:34:37.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zysad6eQ3PIARtveyPfSa787Sgc='} 2024-04-09T23:34:37.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078dd4ad8ad4671f2-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.863 DEBUG:boto:path=/xryqqk-64/ 2024-04-09T23:34:37.863 DEBUG:boto:auth_path=/xryqqk-64/ 2024-04-09T23:34:37.863 DEBUG:boto:Method: HEAD 2024-04-09T23:34:37.863 DEBUG:boto:Path: /xryqqk-64/ 2024-04-09T23:34:37.863 DEBUG:boto:Data: 2024-04-09T23:34:37.863 DEBUG:boto:Headers: {} 2024-04-09T23:34:37.863 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.864 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.864 DEBUG:boto:Params: {} 2024-04-09T23:34:37.864 DEBUG:boto:Token: None 2024-04-09T23:34:37.864 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/ 2024-04-09T23:34:37.864 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-09T23:34:37.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-09T23:34:37.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002efd4809b11462bd-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.868 DEBUG:boto:path=/xryqqk-64/obj185 2024-04-09T23:34:37.869 DEBUG:boto:auth_path=/xryqqk-64/obj185 2024-04-09T23:34:37.869 DEBUG:boto:Method: PUT 2024-04-09T23:34:37.869 DEBUG:boto:Path: /xryqqk-64/obj185 2024-04-09T23:34:37.869 DEBUG:boto:Data: 2024-04-09T23:34:37.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:34:37.869 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:34:37.869 DEBUG:boto:Port: 8000 2024-04-09T23:34:37.869 DEBUG:boto:Params: {} 2024-04-09T23:34:37.869 DEBUG:boto:Token: None 2024-04-09T23:34:37.870 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:34:37 GMT /xryqqk-64/obj185 2024-04-09T23:34:37.870 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:d24UjhFCugwq8imVSlFCHrmoHvI= 2024-04-09T23:34:37.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d24UjhFCugwq8imVSlFCHrmoHvI='} 2024-04-09T23:34:37.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2f64299bf47d9de-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:34:37.876 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-64 2024-04-09T23:34:37.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:34:37.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:34:37.877 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:34:37.933 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:37.933 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:37.969 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-64 2024-04-09T23:34:37.986 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-64 2024-04-09T23:34:37.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:34:37.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:34:37.986 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:34:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:34:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:34:38.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:38.094+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:34:43.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:43.094+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:34:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:48.095+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:34:53.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:53.095+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:34:58.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:34:58.096+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:03.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:03.096+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:08.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:08.097+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:13.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:13.097+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:18.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:18.098+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:23.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:23.099+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:28.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:28.100+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:33.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:33.100+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:38.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:38.100+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:43.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:43.101+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:48.101+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:53.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:53.102+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:35:58.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:35:58.103+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:03.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:03.103+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:08.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:08.104+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:13.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:13.104+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:18.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:18.105+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:23.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:23.105+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:28.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:28.106+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:33.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:33.107+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:38.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:38.107+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:43.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:43.108+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:48.109+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:53.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:53.109+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:36:58.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:36:58.110+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:03.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:03.111+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:08.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:08.112+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:13.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:13.111+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:18.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:18.112+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:23.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:23.112+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:28.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:28.113+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:33.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:33.113+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:38.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:38.114+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:43.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:43.115+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:48.115+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:53.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:53.116+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:37:58.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:37:58.116+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:03.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:03.117+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:08.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:08.118+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:13.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:13.118+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:18.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:18.119+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:23.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:23.120+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:28.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:28.120+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:33.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:33.121+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:38.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:38.122+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:43.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:43.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:48.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:48.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:53.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:53.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:38:58.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:38:58.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:03.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:03.124+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:08.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:08.124+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:13.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:13.125+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:18.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:18.126+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:23.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:23.126+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:28.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:28.127+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-09T23:39:33.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:33.127+0000 7ffa4473b780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:39:33.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:33.127+0000 7ffa4473b780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:39:33.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:33.127+0000 7ffa4473b780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:39:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:39:33.150 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-04-09T23:39:33.152 DEBUG:boto:path=/xryqqk-65/ 2024-04-09T23:39:33.152 DEBUG:boto:auth_path=/xryqqk-65/ 2024-04-09T23:39:33.152 DEBUG:boto:Method: PUT 2024-04-09T23:39:33.152 DEBUG:boto:Path: /xryqqk-65/ 2024-04-09T23:39:33.152 DEBUG:boto:Data: 2024-04-09T23:39:33.152 DEBUG:boto:Headers: {} 2024-04-09T23:39:33.152 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.152 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.153 DEBUG:boto:Params: {} 2024-04-09T23:39:33.153 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:39:33.153 DEBUG:boto:Token: None 2024-04-09T23:39:33.153 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/ 2024-04-09T23:39:33.154 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lXjh9TDaut8lf9TEmUlvLCt6U4c= 2024-04-09T23:39:33.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lXjh9TDaut8lf9TEmUlvLCt6U4c=', 'Content-Length': '0'} 2024-04-09T23:39:33.193 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005364a6309c2437a7-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.194 DEBUG:rgw_multi.tests:created bucket=xryqqk-65 2024-04-09T23:39:33.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:39:33.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:39:33.196 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:39:33.249 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:39:33.250 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:39:33.308 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:39:33.341 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:39:33.190683Z" 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:39:33.342 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:39:33.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:39:33.346 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:39:33.347 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:39:33.347 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:39:33.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:39:33.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:39:33.347 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:39:33.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:39:33.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:39:33.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:33.433+0000 7f7ccfcab780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:39:33.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:33.434+0000 7f7ccfcab780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:39:33.447 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:39:33.448 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.449 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.450 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:39:33.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:39:33.452 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:39:33Z" 2024-04-09T23:39:33.453 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:39:33.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:39:33Z" } 2024-04-09T23:39:33.455 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:39:33.455 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:39:33.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:39:33.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:39:33.455 DEBUG:boto:path=/xryqqk-65/ 2024-04-09T23:39:33.455 DEBUG:boto:auth_path=/xryqqk-65/ 2024-04-09T23:39:33.455 DEBUG:boto:Method: HEAD 2024-04-09T23:39:33.455 DEBUG:boto:Path: /xryqqk-65/ 2024-04-09T23:39:33.456 DEBUG:boto:Data: 2024-04-09T23:39:33.456 DEBUG:boto:Headers: {} 2024-04-09T23:39:33.456 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.456 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.456 DEBUG:boto:Params: {} 2024-04-09T23:39:33.456 DEBUG:boto:Token: None 2024-04-09T23:39:33.456 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/ 2024-04-09T23:39:33.456 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-09T23:39:33.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-09T23:39:33.459 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097ee899dfb62090e-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.459 DEBUG:boto:path=/xryqqk-65/obj11 2024-04-09T23:39:33.459 DEBUG:boto:auth_path=/xryqqk-65/obj11 2024-04-09T23:39:33.459 DEBUG:boto:Method: PUT 2024-04-09T23:39:33.460 DEBUG:boto:Path: /xryqqk-65/obj11 2024-04-09T23:39:33.460 DEBUG:boto:Data: 2024-04-09T23:39:33.460 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:39:33.460 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.460 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.460 DEBUG:boto:Params: {} 2024-04-09T23:39:33.460 DEBUG:boto:Token: None 2024-04-09T23:39:33.460 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/obj11 2024-04-09T23:39:33.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Vac56wNqEZrHpnt6dzTuVrub77E= 2024-04-09T23:39:33.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Vac56wNqEZrHpnt6dzTuVrub77E='} 2024-04-09T23:39:33.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002ddb6fac8b9587f-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.467 DEBUG:boto:path=/xryqqk-65/ 2024-04-09T23:39:33.467 DEBUG:boto:auth_path=/xryqqk-65/ 2024-04-09T23:39:33.467 DEBUG:boto:Method: HEAD 2024-04-09T23:39:33.467 DEBUG:boto:Path: /xryqqk-65/ 2024-04-09T23:39:33.467 DEBUG:boto:Data: 2024-04-09T23:39:33.467 DEBUG:boto:Headers: {} 2024-04-09T23:39:33.467 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.467 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.467 DEBUG:boto:Params: {} 2024-04-09T23:39:33.467 DEBUG:boto:Token: None 2024-04-09T23:39:33.468 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/ 2024-04-09T23:39:33.468 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-09T23:39:33.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-09T23:39:33.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb1ed492ddb3789e-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.470 DEBUG:boto:path=/xryqqk-65/obj22 2024-04-09T23:39:33.470 DEBUG:boto:auth_path=/xryqqk-65/obj22 2024-04-09T23:39:33.470 DEBUG:boto:Method: PUT 2024-04-09T23:39:33.470 DEBUG:boto:Path: /xryqqk-65/obj22 2024-04-09T23:39:33.470 DEBUG:boto:Data: 2024-04-09T23:39:33.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:39:33.471 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.471 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.471 DEBUG:boto:Params: {} 2024-04-09T23:39:33.471 DEBUG:boto:Token: None 2024-04-09T23:39:33.471 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/obj22 2024-04-09T23:39:33.471 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:uFTizUuWJG/M4DPUiSRQTyQtR6s= 2024-04-09T23:39:33.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uFTizUuWJG/M4DPUiSRQTyQtR6s='} 2024-04-09T23:39:33.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce42c06247c16808-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.475 DEBUG:boto:path=/xryqqk-65/ 2024-04-09T23:39:33.475 DEBUG:boto:auth_path=/xryqqk-65/ 2024-04-09T23:39:33.475 DEBUG:boto:Method: HEAD 2024-04-09T23:39:33.475 DEBUG:boto:Path: /xryqqk-65/ 2024-04-09T23:39:33.475 DEBUG:boto:Data: 2024-04-09T23:39:33.475 DEBUG:boto:Headers: {} 2024-04-09T23:39:33.475 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.475 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.476 DEBUG:boto:Params: {} 2024-04-09T23:39:33.476 DEBUG:boto:Token: None 2024-04-09T23:39:33.476 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/ 2024-04-09T23:39:33.476 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-09T23:39:33.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-09T23:39:33.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a2e3098d439cabc-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.478 DEBUG:boto:path=/xryqqk-65/obj33 2024-04-09T23:39:33.478 DEBUG:boto:auth_path=/xryqqk-65/obj33 2024-04-09T23:39:33.478 DEBUG:boto:Method: PUT 2024-04-09T23:39:33.478 DEBUG:boto:Path: /xryqqk-65/obj33 2024-04-09T23:39:33.479 DEBUG:boto:Data: 2024-04-09T23:39:33.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:39:33.479 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.479 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.479 DEBUG:boto:Params: {} 2024-04-09T23:39:33.479 DEBUG:boto:Token: None 2024-04-09T23:39:33.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/obj33 2024-04-09T23:39:33.479 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MWt9sdnJgb5pDKsd28PPqHYx6PY= 2024-04-09T23:39:33.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MWt9sdnJgb5pDKsd28PPqHYx6PY='} 2024-04-09T23:39:33.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048514cbf01c73cfc-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.523 DEBUG:boto:path=/xryqqk-65/ 2024-04-09T23:39:33.523 DEBUG:boto:auth_path=/xryqqk-65/ 2024-04-09T23:39:33.523 DEBUG:boto:Method: HEAD 2024-04-09T23:39:33.523 DEBUG:boto:Path: /xryqqk-65/ 2024-04-09T23:39:33.523 DEBUG:boto:Data: 2024-04-09T23:39:33.523 DEBUG:boto:Headers: {} 2024-04-09T23:39:33.523 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.523 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.523 DEBUG:boto:Params: {} 2024-04-09T23:39:33.524 DEBUG:boto:Token: None 2024-04-09T23:39:33.524 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/ 2024-04-09T23:39:33.524 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-09T23:39:33.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-09T23:39:33.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e7cacf09cb5082-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.526 DEBUG:boto:path=/xryqqk-65/obj44 2024-04-09T23:39:33.527 DEBUG:boto:auth_path=/xryqqk-65/obj44 2024-04-09T23:39:33.527 DEBUG:boto:Method: PUT 2024-04-09T23:39:33.527 DEBUG:boto:Path: /xryqqk-65/obj44 2024-04-09T23:39:33.527 DEBUG:boto:Data: 2024-04-09T23:39:33.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:39:33.527 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.527 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.527 DEBUG:boto:Params: {} 2024-04-09T23:39:33.527 DEBUG:boto:Token: None 2024-04-09T23:39:33.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/obj44 2024-04-09T23:39:33.528 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DZLXM9544MyZdELgD5ow+dZDXoE= 2024-04-09T23:39:33.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DZLXM9544MyZdELgD5ow+dZDXoE='} 2024-04-09T23:39:33.535 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1b2af5f4d685a29-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.535 DEBUG:boto:path=/xryqqk-65/ 2024-04-09T23:39:33.535 DEBUG:boto:auth_path=/xryqqk-65/ 2024-04-09T23:39:33.536 DEBUG:boto:Method: HEAD 2024-04-09T23:39:33.536 DEBUG:boto:Path: /xryqqk-65/ 2024-04-09T23:39:33.536 DEBUG:boto:Data: 2024-04-09T23:39:33.536 DEBUG:boto:Headers: {} 2024-04-09T23:39:33.536 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.536 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.536 DEBUG:boto:Params: {} 2024-04-09T23:39:33.536 DEBUG:boto:Token: None 2024-04-09T23:39:33.536 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/ 2024-04-09T23:39:33.537 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-09T23:39:33.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-09T23:39:33.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031c71aa1462f0d8a-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.539 DEBUG:boto:path=/xryqqk-65/obj55 2024-04-09T23:39:33.539 DEBUG:boto:auth_path=/xryqqk-65/obj55 2024-04-09T23:39:33.539 DEBUG:boto:Method: PUT 2024-04-09T23:39:33.539 DEBUG:boto:Path: /xryqqk-65/obj55 2024-04-09T23:39:33.539 DEBUG:boto:Data: 2024-04-09T23:39:33.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-09T23:39:33.540 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:39:33.540 DEBUG:boto:Port: 8000 2024-04-09T23:39:33.540 DEBUG:boto:Params: {} 2024-04-09T23:39:33.540 DEBUG:boto:Token: None 2024-04-09T23:39:33.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Tue, 09 Apr 2024 23:39:33 GMT /xryqqk-65/obj55 2024-04-09T23:39:33.540 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1c1LE0zRKncrA2Zvg1nrNmdpfZ4= 2024-04-09T23:39:33.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1c1LE0zRKncrA2Zvg1nrNmdpfZ4='} 2024-04-09T23:39:33.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a587b77fcc240055-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:39:33.583 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=xryqqk-65 2024-04-09T23:39:33.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:39:33.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:39:33.583 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:39:33.640 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:39:33.640 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:39:33.695 INFO:teuthology.orchestra.run.smithi063.stdout:Sync is disabled for bucket xryqqk-65 2024-04-09T23:39:33.721 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=xryqqk-65 2024-04-09T23:39:33.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:39:33.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-09T23:39:33.722 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-09T23:39:33.779 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:39:33.779 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:39:33.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:33.830+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:39:38.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:38.831+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:39:43.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:43.831+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:39:48.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:48.832+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:39:53.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:53.832+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:39:58.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:39:58.833+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:03.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:03.834+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:08.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:08.834+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:13.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:13.835+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:18.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:18.836+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:23.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:23.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:28.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:28.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:33.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:33.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:38.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:38.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:43.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:43.838+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:48.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:48.839+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:53.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:53.839+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:40:58.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:40:58.839+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:03.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:03.840+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:08.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:08.841+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:13.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:13.841+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:18.842+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:23.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:23.842+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:28.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:28.843+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:33.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:33.843+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:38.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:38.844+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:43.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:43.844+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:48.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:48.845+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:53.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:53.846+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:41:58.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:41:58.846+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:03.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:03.848+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:08.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:08.848+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:13.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:13.848+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:18.849+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:23.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:23.849+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:28.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:28.850+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:33.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:33.850+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:38.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:38.851+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:43.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:43.852+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:48.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:48.852+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:53.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:53.853+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:42:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:42:58.853+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:03.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:03.854+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:08.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:08.854+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:13.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:13.855+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:18.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:18.855+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:23.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:23.856+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:28.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:28.857+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:33.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:33.858+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:38.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:38.858+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:43.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:43.860+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:48.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:48.859+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:53.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:53.859+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:43:58.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:43:58.860+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:44:03.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:03.861+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:44:08.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:08.861+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:44:13.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:13.862+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:44:18.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:18.862+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:44:23.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:23.863+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-09T23:44:28.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:28.863+0000 7f92802c1780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-09T23:44:28.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:28.863+0000 7f92802c1780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:44:28.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:28.863+0000 7f92802c1780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-09T23:44:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-09T23:44:28.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-04-09T23:44:28.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:28.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:28.885 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:28.938 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:28.939 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:28.994 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:44:29.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:39:33.190683Z" 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:44:29.017 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:29.017 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:44:29.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:29.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:29.018 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:29.067 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.068 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:29.115+0000 7f6a78e5e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:29.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:29.116+0000 7f6a78e5e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:29.132 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:29.133 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:29.134 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:29.135 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:29.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:29.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:29.138 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:29Z" 2024-04-09T23:44:29.138 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:29.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:29Z" } 2024-04-09T23:44:29.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:29.139 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:29.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:29.139 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:44:29.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.140 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.236 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[]} 2024-04-09T23:44:29.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.237 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.287 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.287 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.347 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:44:29.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.348 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.396 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.397 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.474 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:29.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.475 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.587 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-09T23:44:29.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.588 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.633 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.633 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.687 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:29.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.688 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.741 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.741 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.793 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:29.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.794 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:29.917 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:29.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:29.917 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:29.969 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:29.970 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:30.030 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:44:30.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:30.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:30.030 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:30.078 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:30.079 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:30.128 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-09T23:44:30.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:30.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:30.129 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:30.178 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:30.178 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:30.251 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:30.251 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:30.251 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 10, 2024-04-09T23:44:30.251 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:44:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:44:30.253 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:44:30.254 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:44:30.255 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:44:30.256 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:44:30.258 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:44:30.259 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: "directional": [ 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: "source_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: "dest_zone": "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.260 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:44:30.261 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:44:30.262 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:44:30.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.263 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:44:30.263 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:44:30.263 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:44:30.264 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:44:30.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.264 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.264 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "enabled" 2024-04-09T23:44:30.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.265 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:44:30.265 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:44:30.266 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.266 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:44:30.266 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:44:30.267 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:44:30.267 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:44:30.267 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:44:30.267 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:44:30.267 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.268 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:44:30.268 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:44:30.268 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:44:30.268 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:44:30.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.269 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:44:30.269 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:44:30.269 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:44:30.269 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:44:30.269 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.270 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:44:30.270 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:44:30.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:44:30.270 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:44:30.270 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:44:30.271 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:44:30.271 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:44:30.271 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:44:30.271 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:44:30.271 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:44:30.272 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:44:30.273 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.274 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:44:30.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.276 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:44:30.277 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:44:30.277 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.277 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:44:30.279 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:44:30.280 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:44:30.281 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:44:30.281 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:44:30.281 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:44:30.281 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:44:30.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.281 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:44:30.282 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:44:30.282 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:44:30.282 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:44:30.282 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:44:30.282 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:44:30.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:44:30.283 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:30.284 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:30.285 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-66 2024-04-09T23:44:30.285 DEBUG:boto:path=/xryqqk-66/ 2024-04-09T23:44:30.285 DEBUG:boto:auth_path=/xryqqk-66/ 2024-04-09T23:44:30.285 DEBUG:boto:Method: PUT 2024-04-09T23:44:30.285 DEBUG:boto:Path: /xryqqk-66/ 2024-04-09T23:44:30.286 DEBUG:boto:Data: 2024-04-09T23:44:30.286 DEBUG:boto:Headers: {} 2024-04-09T23:44:30.286 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:44:30.286 DEBUG:boto:Port: 8000 2024-04-09T23:44:30.286 DEBUG:boto:Params: {} 2024-04-09T23:44:30.286 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:44:30.287 DEBUG:boto:Token: None 2024-04-09T23:44:30.287 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:44:30 GMT /xryqqk-66/ 2024-04-09T23:44:30.287 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y7h1m1IEbm8/hy8CkkecWh9N0u8= 2024-04-09T23:44:30.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:44:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y7h1m1IEbm8/hy8CkkecWh9N0u8=', 'Content-Length': '0'} 2024-04-09T23:44:30.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e9c87091a58ec07e-006615d2de-21cf-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:44:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:44:30.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:30.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:30.782 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:30.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:30.863 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.017 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:44:31.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.017 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:31.090 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:31.090 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.221 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:31.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.222 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.374 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-09T23:44:31.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.375 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:31.426 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:31.426 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.537 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:31.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.537 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:31.600 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:31.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.702 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:31.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.703 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:31.760 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:31.761 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:44:31.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.832 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:31.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:31.886 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:31.968 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-09T23:44:31.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:44:31.968 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:44:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:32.017 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:32.082 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-04-09T23:44:32.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:32.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:32.083 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:32.131 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:32.131 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:32.175 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.201 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706271.943441_1314.1", 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:44:31.943441Z" 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:32.202 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:44:32.203 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:32.203 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:44:32.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:32.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:32.204 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:32.253 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:32.253 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:32.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:32.306+0000 7fc58d5cf780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:32.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:32.307+0000 7fc58d5cf780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:32.327 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:32.327 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:32.327 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:32.328 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:32.329 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:32.330 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:32.331 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:32.332 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:32Z" 2024-04-09T23:44:32.333 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:32.334 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:32Z" } 2024-04-09T23:44:32.334 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:32.334 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:32.334 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:32.335 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:44:34.586 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:44:34.585+0000 7fc247223700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-09T23:44:37.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:37.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:37.335 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:37.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:37.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:37.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:37.440+0000 7f60f9cf2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:37.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:37.441+0000 7f60f9cf2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:37.463 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:37.463 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:37.464 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:37.465 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:37.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:37.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:37.468 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:37Z" 2024-04-09T23:44:37.469 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:37.470 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:37Z" } 2024-04-09T23:44:37.470 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:37.470 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:37.471 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:37.471 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:44:42.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:42.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:42.472 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:42.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:42.571+0000 7f47bb043780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:42.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:42.572+0000 7f47bb043780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:42.587 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:42.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:42.587 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:42.587 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:42.587 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:42.588 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:42.589 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:42.590 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:42.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:42.592 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:42Z" 2024-04-09T23:44:42.593 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:42.594 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:42Z" } 2024-04-09T23:44:42.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:42.594 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:42.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:42.595 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:44:47.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:47.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:47.596 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:47.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:47.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:47.693+0000 7f8d37759780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:47.694+0000 7f8d37759780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:47.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:47.717 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:47.718 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:47.719 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:47.720 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:47.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:47.722 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:47Z" 2024-04-09T23:44:47.722 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:47.723 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:47Z" } 2024-04-09T23:44:47.723 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:47.723 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:47.723 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:47.723 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:44:52.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:52.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:52.725 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:52.782 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:52.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:52.834+0000 7f16d8b12780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:52.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:52.835+0000 7f16d8b12780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:52.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:52.854 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:52.855 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:52.856 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:52.857 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:52.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:52.859 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:52Z" 2024-04-09T23:44:52.859 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:52.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:52Z" } 2024-04-09T23:44:52.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:52.860 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:52.860 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:52.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:44:57.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:57.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:44:57.862 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:44:57.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:44:57.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:44:57.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:57.972+0000 7f81a8766780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:57.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:44:57.973+0000 7f81a8766780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:44:57.995 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:44:57.995 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:44:57.995 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:44:57.995 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:44:57.995 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:57.996 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:57.997 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:57.998 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:44:57.999 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:58.000 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:44:58.001 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:44:58.001 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:44:58.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:44:58.001 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:44:57Z" 2024-04-09T23:44:58.001 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:44:58.002 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:44:57Z" } 2024-04-09T23:44:58.002 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:44:58.002 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:58.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:44:58.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:03.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:03.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:03.004 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:03.062 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:03.062 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:03.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:03.111+0000 7f5d2bf34780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:03.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:03.112+0000 7f5d2bf34780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:03.127 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:03.127 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:03.127 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:03.127 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:03.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:03.127 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:03.128 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:03.129 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:03.130 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:03.131 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:03.132 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:03.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:03.133 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:03Z" 2024-04-09T23:45:03.133 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:03.134 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:03Z" } 2024-04-09T23:45:03.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:03.134 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:03.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:03.134 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:08.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:08.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:08.136 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:08.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:08.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:08.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:08.230+0000 7f99a1aa8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:08.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:08.231+0000 7f99a1aa8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:08.253 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:08.254 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:08.255 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:08.256 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:08.257 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:08.258 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:08Z" 2024-04-09T23:45:08.259 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:08.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:08Z" } 2024-04-09T23:45:08.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:08.262 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:08.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:08.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:13.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:13.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:13.264 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:13.322 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:13.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:13.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:13.366+0000 7f3c0a6d4780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:13.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:13.367+0000 7f3c0a6d4780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:13.388 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:13.389 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:13.389 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:13.389 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:13.389 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:13.389 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:13.389 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:13.390 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:13.391 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:13.392 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:13.393 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:13.394 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:13.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:13.395 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:13Z" 2024-04-09T23:45:13.395 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:13.396 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:13Z" } 2024-04-09T23:45:13.396 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:13.396 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:13.396 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:13.396 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:18.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:18.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:18.398 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:18.455 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:18.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:18.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:18.496+0000 7fb209d99780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:18.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:18.497+0000 7fb209d99780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:18.513 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:18.513 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:18.513 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:18.513 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:18.513 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:18.513 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:18.514 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:18.515 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:18.516 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:18.517 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:18.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:18.519 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:18.519 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:18.519 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:18.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:18.519 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:18Z" 2024-04-09T23:45:18.519 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:18.520 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:18Z" } 2024-04-09T23:45:18.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:18.520 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:18.520 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:18.521 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:23.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:23.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:23.522 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:23.579 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:23.580 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:23.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:23.627+0000 7f9e993fe780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:23.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:23.628+0000 7f9e993fe780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:23.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:23.644 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:23.645 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:23.646 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:23.647 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:23Z" 2024-04-09T23:45:23.648 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:23.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:23Z" } 2024-04-09T23:45:23.650 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:23.650 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:23.650 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:23.650 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:28.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:28.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:28.651 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:28.709 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:28.709 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:28.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:28.755+0000 7f6955f32780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:28.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:28.757+0000 7f6955f32780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:28.775 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:28.776 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:28.777 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:28.778 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:28.779 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:28.780 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:28Z" 2024-04-09T23:45:28.781 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:28.782 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:28Z" } 2024-04-09T23:45:28.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:28.783 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:28.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:28.783 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:33.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:33.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:33.784 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:33.841 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:33.841 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:33.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:33.880+0000 7fcb1d11c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:33.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:33.881+0000 7fcb1d11c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:33.901 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:33.901 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:33.901 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:33.901 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:33.901 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:33.902 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:33.903 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:33.904 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:33.905 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:33.906 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:33Z" 2024-04-09T23:45:33.907 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:33.908 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:33Z" } 2024-04-09T23:45:33.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:33.909 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:33.909 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:33.909 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:38.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:38.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:38.910 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:38.967 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:38.968 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:39.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:39.015+0000 7f46fb334780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:39.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:39.016+0000 7f46fb334780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:39.031 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:39.031 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:39.031 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:39.032 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:39.033 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:39.034 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:39.035 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:39.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:39.037 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:39.037 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:39.037 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:39.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:39.037 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:39Z" 2024-04-09T23:45:39.037 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:39.038 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:39Z" } 2024-04-09T23:45:39.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:39.038 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:39.038 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:39.039 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:44.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:44.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:44.040 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:44.099 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:44.099 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:44.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:44.140+0000 7fdb8cdc2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:44.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:44.141+0000 7fdb8cdc2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:44.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:44.157 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:44.158 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:44.159 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:44.160 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:44.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:44.162 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:44Z" 2024-04-09T23:45:44.162 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:44.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:44Z" } 2024-04-09T23:45:44.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:44.163 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:44.163 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:44.163 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:49.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:49.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:49.165 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:49.221 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:49.222 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:49.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:49.260+0000 7fdd08575780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:49.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:49.261+0000 7fdd08575780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:49.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:49.280 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:49.281 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:49.282 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712705973.190683_1283.1", 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:49.283 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:49.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:49.285 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:49.285 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:49.285 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:49.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:49.285 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:49Z" 2024-04-09T23:45:49.285 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:49.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712705973.190683_1283.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:33.190683Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:49Z" } 2024-04-09T23:45:49.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:49.286 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:49.286 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:49.286 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712705973.190683_1283.1 2024-04-09T23:45:54.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:54.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:54.288 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:54.342 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:54.342 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:54.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:54.393+0000 7f6f689b2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:54.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:54.394+0000 7f6f689b2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:54.412 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:54.412 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:54.412 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:54.412 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:54.412 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:54.412 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:54.413 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:54.414 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706271.675252_1308.1", 2024-04-09T23:45:54.415 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:44:31.675252Z", 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:54.416 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:54.417 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:54.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:54.418 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:54Z" 2024-04-09T23:45:54.418 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:54.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706271.675252_1308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:44:31.675252Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:54Z" } 2024-04-09T23:45:54.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706271.675252_1308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:44:31.675252Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:54.420 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:54.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.675252_1308.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:54.420 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706271.943441_1314.1 target=1_1712706271.675252_1308.1 2024-04-09T23:45:59.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:59.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:45:59.422 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:45:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:59.490 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:59.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:59.548+0000 7f68167a8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:59.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:45:59.549+0000 7f68167a8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:45:59.571 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:45:59.571 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:45:59.571 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:45:59.571 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:45:59.571 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:45:59.572 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:59.573 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:59.574 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706271.943441_1314.1", 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:44:31.943441Z", 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:45:59.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:59.576 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:45:59.577 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:45:59.577 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:45:59.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:45:59.577 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:45:59Z" 2024-04-09T23:45:59.577 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:45:59.578 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706271.943441_1314.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:44:31.943441Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:45:59Z" } 2024-04-09T23:45:59.578 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706271.943441_1314.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:44:31.943441Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:45:59.578 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:59.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:45:59.578 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:45:59.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.579 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:45:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:59.687 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:45:59.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.688 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:45:59.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:59.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:59.822 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:45:59.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.823 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:45:59.878 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:45:59.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:45:59.959 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-09T23:45:59.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:45:59.960 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:00.009 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:00.009 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:00.065 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-09T23:46:00.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:00.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xryqqk-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:00.066 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xryqqk-66 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:00.114 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:00.114 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:00.186 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[]} 2024-04-09T23:46:00.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:00.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:00.187 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:00.234 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:00.235 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:00.291 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:46:00.315 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:46:00.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:46:00.051842Z" 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.317 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:00.317 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:46:00.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:00.317 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:46:00.318 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:00.318 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:46:00.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:00.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:00.318 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:00.367 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:00.367 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:00.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:00.402+0000 7fae24dd5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:00.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:00.403+0000 7fae24dd5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.417 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:00.418 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:00.419 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706271.943441_1314.1", 2024-04-09T23:46:00.420 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:44:31.943441Z", 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:00.421 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:00.422 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:00Z" 2024-04-09T23:46:00.423 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:00.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706271.943441_1314.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:44:31.943441Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:00Z" } 2024-04-09T23:46:00.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706271.943441_1314.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:44:31.943441Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:00.425 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:00.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:00.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706360.051842_1327.1 target=1_1712706271.943441_1314.1 2024-04-09T23:46:05.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:05.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:05.427 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:05.484 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:05.484 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:05.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:05.526+0000 7fb121edc780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:05.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:05.527+0000 7fb121edc780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:05.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:05.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:05.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706271.943441_1314.1", 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:44:31.943441Z", 2024-04-09T23:46:05.545 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:05.546 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:05Z" 2024-04-09T23:46:05.547 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:05.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706271.943441_1314.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:44:31.943441Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:05Z" } 2024-04-09T23:46:05.549 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706271.943441_1314.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:44:31.943441Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:05.549 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:05.549 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:05.549 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706360.051842_1327.1 target=1_1712706271.943441_1314.1 2024-04-09T23:46:10.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:10.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:10.551 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:10.608 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:10.608 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:10.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:10.653+0000 7f1f95491780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:10.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:10.654+0000 7f1f95491780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:10.677 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:10.677 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:10.677 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:10.677 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:10.678 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:10.679 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:10.680 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706271.943441_1314.1", 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:44:31.943441Z", 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:10.681 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:10.682 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:10.683 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:10.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:10.683 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:10Z" 2024-04-09T23:46:10.683 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:10.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706271.943441_1314.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:44:31.943441Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:10Z" } 2024-04-09T23:46:10.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706271.943441_1314.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:44:31.943441Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:10.684 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:10.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706271.943441_1314.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:10.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706360.051842_1327.1 target=1_1712706271.943441_1314.1 2024-04-09T23:46:15.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:15.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:15.686 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:15.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:15.744 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:15.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:15.797+0000 7fd19350f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:15.798+0000 7fd19350f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:15Z" 2024-04-09T23:46:15.819 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:15.820 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:15Z" } 2024-04-09T23:46:15.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:15.820 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:15.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:15.820 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:46:15.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:15.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:15.821 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:15.872 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:15.872 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:15.924 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:46:15.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:15.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:15.925 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:15.980 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:15.980 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.052 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:46:16.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.052 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:16.106 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.106 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.179 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-09T23:46:16.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.180 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:16.226 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.226 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.279 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-09T23:46:16.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.279 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:16.324 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.324 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.387 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[]} 2024-04-09T23:46:16.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.387 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:16.436 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.436 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.514 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 11, 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:46:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:46:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:46:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:46:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:46:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:46:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:46:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:46:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:46:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:16.542 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:16.543 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-04-09T23:46:16.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:16.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:16.545 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:16.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.650 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:46:00.051842Z" 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.701 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:46:16.702 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:16.702 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:46:16.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:16.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:16.703 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:16.766 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.766 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:16.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:16.843+0000 7f7cf7069780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:16.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:16.844+0000 7f7cf7069780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:16Z" 2024-04-09T23:46:16.902 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:16.903 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:16Z" } 2024-04-09T23:46:16.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:16.903 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:16.903 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:16.903 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:46:16.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:16.904 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:16.974 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:16.974 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:17.177 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[]} 2024-04-09T23:46:17.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.178 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:17.244 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:17.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:17.376 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:46:17.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.377 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:17.441 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:17.442 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:17.546 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-09T23:46:17.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.547 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:17.608 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:17.609 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:17.709 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:46:17.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.710 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:17.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:17.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:17.835 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:46:17.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.836 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:17.884 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:17.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:17.962 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:17.962 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:17.962 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 12, 2024-04-09T23:46:17.962 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:46:17.962 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:46:17.962 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:46:17.963 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:17.964 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:17.965 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:17.966 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:17.967 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:46:17.968 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:46:17.969 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:46:17.972 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "enabled" 2024-04-09T23:46:17.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:46:17.974 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:46:17.975 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:17.976 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:46:17.977 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.978 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:46:17.979 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:46:17.980 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:46:17.981 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:46:17.982 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:46:17.983 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:46:17.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:17.985 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:46:17.986 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:46:17.986 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:17.986 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:17.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:46:17.987 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:46:18.033 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:18.033 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:18.105 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:46:18.106 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-67 2024-04-09T23:46:18.106 DEBUG:boto:path=/xryqqk-67/ 2024-04-09T23:46:18.106 DEBUG:boto:auth_path=/xryqqk-67/ 2024-04-09T23:46:18.106 DEBUG:boto:Method: PUT 2024-04-09T23:46:18.106 DEBUG:boto:Path: /xryqqk-67/ 2024-04-09T23:46:18.106 DEBUG:boto:Data: 2024-04-09T23:46:18.106 DEBUG:boto:Headers: {} 2024-04-09T23:46:18.106 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:46:18.107 DEBUG:boto:Port: 8000 2024-04-09T23:46:18.107 DEBUG:boto:Params: {} 2024-04-09T23:46:18.107 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:46:18.107 DEBUG:boto:Token: None 2024-04-09T23:46:18.108 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:46:18 GMT /xryqqk-67/ 2024-04-09T23:46:18.108 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Mp7LSQX3BdyIXXnd5f6NMMW2Qew= 2024-04-09T23:46:18.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mp7LSQX3BdyIXXnd5f6NMMW2Qew=', 'Content-Length': '0'} 2024-04-09T23:46:18.313 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cf651e4fb1c0d6b3-006615d34a-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:46:18.314 DEBUG:boto:path=/xryqqk-67/ 2024-04-09T23:46:18.314 DEBUG:boto:auth_path=/xryqqk-67/ 2024-04-09T23:46:18.314 DEBUG:boto:Method: HEAD 2024-04-09T23:46:18.314 DEBUG:boto:Path: /xryqqk-67/ 2024-04-09T23:46:18.314 DEBUG:boto:Data: 2024-04-09T23:46:18.314 DEBUG:boto:Headers: {} 2024-04-09T23:46:18.315 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:46:18.315 DEBUG:boto:Port: 8000 2024-04-09T23:46:18.315 DEBUG:boto:Params: {} 2024-04-09T23:46:18.315 DEBUG:boto:Token: None 2024-04-09T23:46:18.315 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:46:18 GMT /xryqqk-67/ 2024-04-09T23:46:18.315 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5LV9wr2y/I8NI5WmVjC0L0Dxr6Q= 2024-04-09T23:46:18.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5LV9wr2y/I8NI5WmVjC0L0Dxr6Q=', 'Content-Length': '0'} 2024-04-09T23:46:18.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a7bdb1688ec4201-006615d34a-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:46:18.319 DEBUG:boto:path=/xryqqk-67/obj1 2024-04-09T23:46:18.319 DEBUG:boto:auth_path=/xryqqk-67/obj1 2024-04-09T23:46:18.319 DEBUG:boto:Method: PUT 2024-04-09T23:46:18.319 DEBUG:boto:Path: /xryqqk-67/obj1 2024-04-09T23:46:18.319 DEBUG:boto:Data: 2024-04-09T23:46:18.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:46:18.320 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:46:18.320 DEBUG:boto:Port: 8000 2024-04-09T23:46:18.320 DEBUG:boto:Params: {} 2024-04-09T23:46:18.320 DEBUG:boto:Token: None 2024-04-09T23:46:18.320 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:46:18 GMT /xryqqk-67/obj1 2024-04-09T23:46:18.320 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bjQcZaoP0t4MxEs25qXdmiay420= 2024-04-09T23:46:18.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bjQcZaoP0t4MxEs25qXdmiay420='} 2024-04-09T23:46:18.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000205e7f7b368d6f0d-006615d34a-237a-a1'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:46:18.340 INFO:rgw_multi.tests:create bucket zone=a2 name=xryqqk-68 2024-04-09T23:46:18.340 DEBUG:boto:path=/xryqqk-68/ 2024-04-09T23:46:18.341 DEBUG:boto:auth_path=/xryqqk-68/ 2024-04-09T23:46:18.341 DEBUG:boto:Method: PUT 2024-04-09T23:46:18.341 DEBUG:boto:Path: /xryqqk-68/ 2024-04-09T23:46:18.341 DEBUG:boto:Data: 2024-04-09T23:46:18.341 DEBUG:boto:Headers: {} 2024-04-09T23:46:18.341 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:46:18.341 DEBUG:boto:Port: 8001 2024-04-09T23:46:18.341 DEBUG:boto:Params: {} 2024-04-09T23:46:18.342 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:46:18.342 DEBUG:boto:Token: None 2024-04-09T23:46:18.342 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:46:18 GMT /xryqqk-68/ 2024-04-09T23:46:18.342 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Ajb480v8fV8PW0zYIYdLldWQtr8= 2024-04-09T23:46:18.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ajb480v8fV8PW0zYIYdLldWQtr8=', 'Content-Length': '0'} 2024-04-09T23:46:18.467 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000057943f7a2d5934e-006615d34a-237c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:46:18.467 DEBUG:boto:path=/xryqqk-68/ 2024-04-09T23:46:18.467 DEBUG:boto:auth_path=/xryqqk-68/ 2024-04-09T23:46:18.468 DEBUG:boto:Method: HEAD 2024-04-09T23:46:18.468 DEBUG:boto:Path: /xryqqk-68/ 2024-04-09T23:46:18.468 DEBUG:boto:Data: 2024-04-09T23:46:18.468 DEBUG:boto:Headers: {} 2024-04-09T23:46:18.468 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:46:18.468 DEBUG:boto:Port: 8001 2024-04-09T23:46:18.468 DEBUG:boto:Params: {} 2024-04-09T23:46:18.468 DEBUG:boto:Token: None 2024-04-09T23:46:18.469 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:46:18 GMT /xryqqk-68/ 2024-04-09T23:46:18.469 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5z8F3SDMZv8phEaEFnpLRB0SMQg= 2024-04-09T23:46:18.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5z8F3SDMZv8phEaEFnpLRB0SMQg=', 'Content-Length': '0'} 2024-04-09T23:46:18.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7ba13c4d3eb562f-006615d34a-237c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:46:18.471 DEBUG:boto:path=/xryqqk-68/obj2 2024-04-09T23:46:18.471 DEBUG:boto:auth_path=/xryqqk-68/obj2 2024-04-09T23:46:18.472 DEBUG:boto:Method: PUT 2024-04-09T23:46:18.472 DEBUG:boto:Path: /xryqqk-68/obj2 2024-04-09T23:46:18.472 DEBUG:boto:Data: 2024-04-09T23:46:18.472 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:46:18.472 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:46:18.472 DEBUG:boto:Port: 8001 2024-04-09T23:46:18.472 DEBUG:boto:Params: {} 2024-04-09T23:46:18.472 DEBUG:boto:Token: None 2024-04-09T23:46:18.472 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:46:18 GMT /xryqqk-68/obj2 2024-04-09T23:46:18.473 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xI6TCrp+gCqCoivrbT7cPbPesek= 2024-04-09T23:46:18.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xI6TCrp+gCqCoivrbT7cPbPesek='} 2024-04-09T23:46:18.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001009301bd6468e31-006615d34a-237c-a2'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:46:18.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:18.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:18.496 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:18.553 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:18.553 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:18.631 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:46:18.667 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:46:18.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706378.392455_1368.1", 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:46:18.392455Z" 2024-04-09T23:46:18.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.669 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:18.669 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:46:18.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:18.669 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:46:18.670 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:18.670 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:46:18.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:18.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:18.670 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:18.725 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:18.725 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:18.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:18.789+0000 7f840c3a1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:18.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:18.791+0000 7f840c3a1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:18.820 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:18.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:18.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:18.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:18.824 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:18.825 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:18Z" 2024-04-09T23:46:18.826 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:18.827 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:18Z" } 2024-04-09T23:46:18.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:18.828 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:18.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:18.828 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:23.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:23.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:23.830 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:23.887 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:23.887 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:23.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:23.935+0000 7fc04804a780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:23.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:23.936+0000 7fc04804a780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:23.958 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:23.958 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:23.959 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:23.960 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:23.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:23.962 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:23.963 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:23.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:23.965 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:23Z" 2024-04-09T23:46:23.965 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:23.966 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:23Z" } 2024-04-09T23:46:23.966 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:23.966 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:23.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:23.966 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:28.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:28.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:28.968 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:29.025 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:29.025 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:29.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:29.073+0000 7f1578cfa780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:29.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:29.074+0000 7f1578cfa780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:29.089 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:29.089 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:29.090 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:29.091 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:29.092 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:29.093 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:29.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:29.095 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:29.095 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:29.095 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:29.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:29.095 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:29Z" 2024-04-09T23:46:29.095 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:29.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:29Z" } 2024-04-09T23:46:29.097 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:29.097 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:29.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:29.097 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:34.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:34.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:34.099 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:34.157 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:34.158 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:34.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:34.216+0000 7f9703467780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:34.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:34.217+0000 7f9703467780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:34.238 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:34.239 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:34.240 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:34.241 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:34.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:34.243 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:34Z" 2024-04-09T23:46:34.244 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:34.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:34Z" } 2024-04-09T23:46:34.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:34.245 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:34.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:34.245 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:39.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:39.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:39.247 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:39.304 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:39.304 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:39.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:39.352+0000 7f4edde9e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:39.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:39.353+0000 7f4edde9e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:39.378 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:39.378 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:39.379 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:39.380 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:39.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:39.382 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:39.383 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:39.384 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:39.385 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:39Z" 2024-04-09T23:46:39.385 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:39.387 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:39Z" } 2024-04-09T23:46:39.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:39.387 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:39.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:39.388 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:44.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:44.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:44.389 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:44.445 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:44.445 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:44.490+0000 7fb191248780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:44.491+0000 7fb191248780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:44.507 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:44.507 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:44.507 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:44.508 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:44.509 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:44.510 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:44.511 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:44.512 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:44Z" 2024-04-09T23:46:44.513 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:44.514 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:44Z" } 2024-04-09T23:46:44.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:44.514 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:44.515 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:44.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:49.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:49.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:49.516 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:49.624+0000 7fbf456ae780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:49.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:49.625+0000 7fbf456ae780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:49.642 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:49.642 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:49.642 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:49.642 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:49.643 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:49.645 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:49.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:49.647 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:49Z" 2024-04-09T23:46:49.648 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:49.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:49Z" } 2024-04-09T23:46:49.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:49.649 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:49.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:49.650 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:54.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:54.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:54.651 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:54.710 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:54.710 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:54.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:54.757+0000 7f3636761780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:54.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:54.758+0000 7f3636761780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:54.782 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:54.782 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:54.783 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:54.784 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:54.785 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:54.786 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:54.788 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:54Z" 2024-04-09T23:46:54.788 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:54.788 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:54Z" } 2024-04-09T23:46:54.788 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:54.788 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:54.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:54.789 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:46:59.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:59.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:46:59.790 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:46:59.847 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:46:59.848 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:46:59.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:59.890+0000 7f8fdbcda780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:59.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:46:59.891+0000 7f8fdbcda780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:46:59.904 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:46:59.904 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:59.905 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:59.906 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:59.907 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:46:59.908 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:59.909 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:46:59.910 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:46:59.910 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:46:59.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:46:59.910 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:46:59Z" 2024-04-09T23:46:59.910 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:46:59.911 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:46:59Z" } 2024-04-09T23:46:59.911 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:46:59.911 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:59.911 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:46:59.911 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:04.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:04.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:04.913 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:04.971 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:04.971 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:05.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:05.012+0000 7f2b4cb3a780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:05.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:05.013+0000 7f2b4cb3a780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:05.026 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:05.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:05.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:05.029 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:05.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:05.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:05Z" 2024-04-09T23:47:05.032 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:05.033 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:05Z" } 2024-04-09T23:47:05.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:05.034 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:05.034 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:05.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:10.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:10.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:10.035 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:10.093 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:10.094 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:10.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:10.135+0000 7f8abd3c1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:10.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:10.136+0000 7f8abd3c1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:10.176 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:10.176 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:10.177 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:10.178 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:10.179 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:10.180 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:10.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:10Z" 2024-04-09T23:47:10.182 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:10.183 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:10Z" } 2024-04-09T23:47:10.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:10.183 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:10.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:10.184 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:15.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:15.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:15.185 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:15.242 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:15.242 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:15.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:15.294+0000 7f502002f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:15.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:15.295+0000 7f502002f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:15.313 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:15.313 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:15.313 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:15.314 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:15.315 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:15.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:15.317 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:15.318 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:15Z" 2024-04-09T23:47:15.319 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:15.320 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:15Z" } 2024-04-09T23:47:15.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:15.320 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:15.321 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:15.321 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:20.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:20.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:20.322 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:20.379 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:20.379 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:20.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:20.420+0000 7f09e997d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:20.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:20.421+0000 7f09e997d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:20.443 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:20.444 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:20.445 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:20.446 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:20.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:20.448 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:20Z" 2024-04-09T23:47:20.449 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:20.450 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:20Z" } 2024-04-09T23:47:20.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:20.451 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:20.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:20.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:25.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:25.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:25.452 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:25.510 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:25.510 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:25.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:25.563+0000 7f1a2b88b780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:25.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:25.564+0000 7f1a2b88b780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:25.585 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:25.586 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:25.587 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:25.588 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:25.589 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:25Z" 2024-04-09T23:47:25.589 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:25.590 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:25Z" } 2024-04-09T23:47:25.590 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:25.590 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:25.590 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:25.590 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:30.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:30.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:30.591 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:30.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:30.649 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:30.689+0000 7f728d075780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:30.690+0000 7f728d075780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:30.706 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:30.706 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:30Z" 2024-04-09T23:47:30.713 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:30.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:30Z" } 2024-04-09T23:47:30.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:30.714 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:30.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:30.714 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:35.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:35.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:35.716 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:35.772 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:35.772 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:35.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:35.823+0000 7fa70c4e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:35.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:35.824+0000 7fa70c4e6780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:35.843 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:35.843 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:35.844 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:35.845 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:35Z" 2024-04-09T23:47:35.848 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:35.849 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:35Z" } 2024-04-09T23:47:35.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:35.850 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:35.850 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:35.850 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:40.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:40.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:40.851 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:40.951+0000 7fafebfae780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:40.952+0000 7fafebfae780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:40.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:40.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:40.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:40Z" 2024-04-09T23:47:40.975 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:40.976 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:40Z" } 2024-04-09T23:47:40.976 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:40.976 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:40.976 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:40.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:45.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:45.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:45.977 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:46.035 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:46.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:46.077+0000 7f8f831fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:46.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:46.078+0000 7f8f831fd780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:46.098 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:46.098 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:46.098 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:46.098 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:46.098 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:46.098 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:46.099 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:46.100 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:46.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:46.103 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:46.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:46.105 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:46Z" 2024-04-09T23:47:46.105 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:46.106 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:46Z" } 2024-04-09T23:47:46.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:46.106 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:46.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:46.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:49.350 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-09T23:47:49.350+0000 7fc24521f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-09T23:47:51.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:51.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:51.108 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:51.165 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:51.166 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:51.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:51.213+0000 7fe379bab780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:51.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:51.214+0000 7fe379bab780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:51.846 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:51.846 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:51.847 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:51.848 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:51.849 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:51.850 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:51.851 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:51.852 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:51.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:51.852 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:51Z" 2024-04-09T23:47:51.852 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:51.852 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:51Z" } 2024-04-09T23:47:51.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:51.852 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:51.852 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:51.853 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:47:56.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:56.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:47:56.855 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:47:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:47:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:47:56.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:56.970+0000 7fee4635e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:56.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:47:56.971+0000 7fee4635e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:47:56.989 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:47:56.989 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:47:56.989 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:47:56.989 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:56.990 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:56.991 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:56.992 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706360.051842_1327.1", 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:47:56.993 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:47:56.994 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:47:56Z" 2024-04-09T23:47:56.995 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:47:56.996 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706360.051842_1327.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:00.051842Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:47:56Z" } 2024-04-09T23:47:56.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:47:56.996 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:56.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:47:56.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-09T23:48:01.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:01.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:01.998 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:02.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:02.109+0000 7f1e20a2c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:02.110+0000 7f1e20a2c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:02.126 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:02.126 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.127 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.128 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.129 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706378.392455_1368.1", 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.392455Z", 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:02Z" 2024-04-09T23:48:02.132 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:02.133 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706378.392455_1368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:18.392455Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:48:02Z" } 2024-04-09T23:48:02.133 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706378.392455_1368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.392455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:48:02.134 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:02.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:02.134 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:48:02.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.134 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:02.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:02.248 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.249 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:48:02.250 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:02.250 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:48:02.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.367 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:02.367 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.368 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:02.369 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.370 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.371 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:02.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.373 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:02.373 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:02.373 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:02.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.373 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:02Z" 2024-04-09T23:48:02.373 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:02.374 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:02Z" } 2024-04-09T23:48:02.374 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:02.374 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:02.374 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:02.375 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:48:02.375 DEBUG:boto:path=/xryqqk-67/ 2024-04-09T23:48:02.375 DEBUG:boto:auth_path=/xryqqk-67/ 2024-04-09T23:48:02.375 DEBUG:boto:Method: HEAD 2024-04-09T23:48:02.375 DEBUG:boto:Path: /xryqqk-67/ 2024-04-09T23:48:02.376 DEBUG:boto:Data: 2024-04-09T23:48:02.376 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.376 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:48:02.376 DEBUG:boto:Port: 8001 2024-04-09T23:48:02.376 DEBUG:boto:Params: {} 2024-04-09T23:48:02.376 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:48:02.377 DEBUG:boto:Token: None 2024-04-09T23:48:02.377 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-67/ 2024-04-09T23:48:02.377 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wpzBXAA/E1N9VMbaHWV0KHYrpzw= 2024-04-09T23:48:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wpzBXAA/E1N9VMbaHWV0KHYrpzw=', 'Content-Length': '0'} 2024-04-09T23:48:02.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fdb938b71fb67ed-006615d3b2-237c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.382 DEBUG:boto:path=/xryqqk-67/obj1 2024-04-09T23:48:02.382 DEBUG:boto:auth_path=/xryqqk-67/obj1 2024-04-09T23:48:02.382 DEBUG:boto:Method: HEAD 2024-04-09T23:48:02.382 DEBUG:boto:Path: /xryqqk-67/obj1 2024-04-09T23:48:02.382 DEBUG:boto:Data: 2024-04-09T23:48:02.383 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.383 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:48:02.383 DEBUG:boto:Port: 8001 2024-04-09T23:48:02.383 DEBUG:boto:Params: {} 2024-04-09T23:48:02.383 DEBUG:boto:Token: None 2024-04-09T23:48:02.383 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-67/obj1 2024-04-09T23:48:02.383 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SOizCtI6mO5Cj7q+tYaYy+HU3FQ= 2024-04-09T23:48:02.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SOizCtI6mO5Cj7q+tYaYy+HU3FQ=', 'Content-Length': '0'} 2024-04-09T23:48:02.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-67:a5723776-f142-428c-a6df-d7bf949257da.9086.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000178fb8a8db0a35ab-006615d3b2-237c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.385 DEBUG:boto:path=/xryqqk-67/obj1 2024-04-09T23:48:02.385 DEBUG:boto:auth_path=/xryqqk-67/obj1 2024-04-09T23:48:02.385 DEBUG:boto:Method: GET 2024-04-09T23:48:02.385 DEBUG:boto:Path: /xryqqk-67/obj1 2024-04-09T23:48:02.386 DEBUG:boto:Data: 2024-04-09T23:48:02.386 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.386 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:48:02.386 DEBUG:boto:Port: 8001 2024-04-09T23:48:02.386 DEBUG:boto:Params: {} 2024-04-09T23:48:02.386 DEBUG:boto:Token: None 2024-04-09T23:48:02.386 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-67/obj1 2024-04-09T23:48:02.386 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IKXVa/ihegDgLRkYfQhHmw+Swtk= 2024-04-09T23:48:02.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IKXVa/ihegDgLRkYfQhHmw+Swtk=', 'Content-Length': '0'} 2024-04-09T23:48:02.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-67:a5723776-f142-428c-a6df-d7bf949257da.9086.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000807a3de531046285-006615d3b2-237c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.388 DEBUG:boto:path=/xryqqk-68/ 2024-04-09T23:48:02.388 DEBUG:boto:auth_path=/xryqqk-68/ 2024-04-09T23:48:02.389 DEBUG:boto:Method: HEAD 2024-04-09T23:48:02.389 DEBUG:boto:Path: /xryqqk-68/ 2024-04-09T23:48:02.389 DEBUG:boto:Data: 2024-04-09T23:48:02.389 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.389 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:02.389 DEBUG:boto:Port: 8000 2024-04-09T23:48:02.389 DEBUG:boto:Params: {} 2024-04-09T23:48:02.389 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:48:02.389 DEBUG:boto:Token: None 2024-04-09T23:48:02.389 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-68/ 2024-04-09T23:48:02.390 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ImcKSFb6/Bcq/KZCmQojtxdjWTM= 2024-04-09T23:48:02.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ImcKSFb6/Bcq/KZCmQojtxdjWTM=', 'Content-Length': '0'} 2024-04-09T23:48:02.393 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad550daca097f858-006615d3b2-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.393 DEBUG:boto:path=/xryqqk-68/obj2 2024-04-09T23:48:02.393 DEBUG:boto:auth_path=/xryqqk-68/obj2 2024-04-09T23:48:02.394 DEBUG:boto:Method: HEAD 2024-04-09T23:48:02.394 DEBUG:boto:Path: /xryqqk-68/obj2 2024-04-09T23:48:02.394 DEBUG:boto:Data: 2024-04-09T23:48:02.394 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.394 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:02.394 DEBUG:boto:Port: 8000 2024-04-09T23:48:02.394 DEBUG:boto:Params: {} 2024-04-09T23:48:02.394 DEBUG:boto:Token: None 2024-04-09T23:48:02.394 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-68/obj2 2024-04-09T23:48:02.394 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:BE92Ab5VMSrpOZ3ytQgJpCYHlTQ= 2024-04-09T23:48:02.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BE92Ab5VMSrpOZ3ytQgJpCYHlTQ=', 'Content-Length': '0'} 2024-04-09T23:48:02.396 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d04349a6a226e3bd-006615d3b2-237a-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.398 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-04-09T23:48:02.399 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-09T23:48:02.400 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-09T23:48:02.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.400 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:02.450 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.450 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.494 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:48:02.508 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:48:02.508 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706378.392455_1368.1", 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:46:18.392455Z" 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.509 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.510 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:02.510 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:48:02.510 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.510 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:48:02.511 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:02.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:48:02.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.511 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:02.560 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.560 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:02.593+0000 7f36bf180780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:02.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:02.594+0000 7f36bf180780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:02.609 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:02.609 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.610 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.611 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.612 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706378.392455_1368.1", 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.392455Z", 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:02.613 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.614 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:02.615 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:48:02.615 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:48:02.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:02.615 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:02Z" 2024-04-09T23:48:02.615 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:02.616 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706378.392455_1368.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:46:18.392455Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:48:02Z" } 2024-04-09T23:48:02.616 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706378.392455_1368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.392455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:48:02.616 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:02.616 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:02.616 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:48:02.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:02.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:02.617 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:02.662 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.662 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.735 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:48:02.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:02.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:02.736 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:02.781 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.781 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.857 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:48:02.857 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-69 2024-04-09T23:48:02.857 DEBUG:boto:path=/xryqqk-69/ 2024-04-09T23:48:02.857 DEBUG:boto:auth_path=/xryqqk-69/ 2024-04-09T23:48:02.857 DEBUG:boto:Method: PUT 2024-04-09T23:48:02.857 DEBUG:boto:Path: /xryqqk-69/ 2024-04-09T23:48:02.858 DEBUG:boto:Data: 2024-04-09T23:48:02.858 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.858 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:02.858 DEBUG:boto:Port: 8000 2024-04-09T23:48:02.858 DEBUG:boto:Params: {} 2024-04-09T23:48:02.858 DEBUG:boto:Token: None 2024-04-09T23:48:02.858 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-69/ 2024-04-09T23:48:02.858 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/l7EIfrYi4gEa24JvsFfp8ueS6c= 2024-04-09T23:48:02.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/l7EIfrYi4gEa24JvsFfp8ueS6c=', 'Content-Length': '0'} 2024-04-09T23:48:02.880 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ddadec9e145ef531-006615d3b2-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.880 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-70 2024-04-09T23:48:02.881 DEBUG:boto:path=/xryqqk-70/ 2024-04-09T23:48:02.881 DEBUG:boto:auth_path=/xryqqk-70/ 2024-04-09T23:48:02.881 DEBUG:boto:Method: PUT 2024-04-09T23:48:02.881 DEBUG:boto:Path: /xryqqk-70/ 2024-04-09T23:48:02.881 DEBUG:boto:Data: 2024-04-09T23:48:02.881 DEBUG:boto:Headers: {} 2024-04-09T23:48:02.881 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:02.881 DEBUG:boto:Port: 8000 2024-04-09T23:48:02.881 DEBUG:boto:Params: {} 2024-04-09T23:48:02.882 DEBUG:boto:Token: None 2024-04-09T23:48:02.882 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:48:02 GMT /xryqqk-70/ 2024-04-09T23:48:02.882 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ce05M61rBsndAHHRmvdXflDENZ0= 2024-04-09T23:48:02.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ce05M61rBsndAHHRmvdXflDENZ0=', 'Content-Length': '0'} 2024-04-09T23:48:02.899 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000619a293a99aa86c4-006615d3b2-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:02.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:02.900 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:02.954 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:02.954 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:02.998 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:03.016 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706482.898476_1388.1", 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:02.898476Z" 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.017 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:48:03.018 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:03.018 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:48:03.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:03.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:03.019 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:03.068 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.068 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:03.124+0000 7f0c827e2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:03.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:03.126+0000 7f0c827e2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:03.147 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:03.148 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:03.149 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.150 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706482.898476_1388.1", 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-09T23:48:03.151 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:03.152 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:03Z" 2024-04-09T23:48:03.153 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:03.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706482.898476_1388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:02.898476Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:48:03Z" } 2024-04-09T23:48:03.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706482.898476_1388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:02.898476Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:48:03.155 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:03.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:03.155 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:48:03.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.155 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.261 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:48:03.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.262 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.370 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:48:03.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=xryqqk-69', '--dest-bucket=xryqqk-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=xryqqk-69', '--dest-bucket=xryqqk-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.370 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=xryqqk-69 --dest-bucket=xryqqk-69 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.503 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:48:03.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.503 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.628 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:48:03.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.629 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.676 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.676 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.728 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:48:03.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.729 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.778 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.779 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:03.849 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:03.849 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:48:03.849 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 13, 2024-04-09T23:48:03.849 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:48:03.850 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.851 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:48:03.852 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.853 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:48:03.854 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.855 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:48:03.856 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:48:03.857 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-69", 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-69", 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:48:03.858 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:48:03.859 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "enabled" 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.860 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:48:03.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:48:03.862 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:48:03.863 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:48:03.864 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:48:03.865 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:48:03.866 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.867 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.877 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:03.878 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.879 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:48:03.880 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:48:03.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:03.882 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:03.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:48:03.884 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:48:03.932 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:03.933 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:04.019 INFO:teuthology.orchestra.run.smithi063.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"xryqqk-69"},"dest":{"zone":"a1","bucket":"xryqqk-69"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"xryqqk-69"},"dest":{"zone":"a2","bucket":"xryqqk-69"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-09T23:48:04.020 DEBUG:boto:path=/xryqqk-69/ 2024-04-09T23:48:04.020 DEBUG:boto:auth_path=/xryqqk-69/ 2024-04-09T23:48:04.020 DEBUG:boto:Method: HEAD 2024-04-09T23:48:04.020 DEBUG:boto:Path: /xryqqk-69/ 2024-04-09T23:48:04.020 DEBUG:boto:Data: 2024-04-09T23:48:04.020 DEBUG:boto:Headers: {} 2024-04-09T23:48:04.021 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.021 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.021 DEBUG:boto:Params: {} 2024-04-09T23:48:04.021 DEBUG:boto:Token: None 2024-04-09T23:48:04.021 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-69/ 2024-04-09T23:48:04.021 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo= 2024-04-09T23:48:04.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo=', 'Content-Length': '0'} 2024-04-09T23:48:04.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e91a8df5f85901c-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.102 DEBUG:boto:path=/xryqqk-69/obj1 2024-04-09T23:48:04.102 DEBUG:boto:auth_path=/xryqqk-69/obj1 2024-04-09T23:48:04.102 DEBUG:boto:Method: PUT 2024-04-09T23:48:04.102 DEBUG:boto:Path: /xryqqk-69/obj1 2024-04-09T23:48:04.102 DEBUG:boto:Data: 2024-04-09T23:48:04.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:48:04.102 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.102 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.102 DEBUG:boto:Params: {} 2024-04-09T23:48:04.103 DEBUG:boto:Token: None 2024-04-09T23:48:04.103 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-69/obj1 2024-04-09T23:48:04.103 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:09VwPBeTCM99Ds9Cq2u5kFp/AtQ= 2024-04-09T23:48:04.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:09VwPBeTCM99Ds9Cq2u5kFp/AtQ='} 2024-04-09T23:48:04.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a39c695455254ad-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.120 DEBUG:boto:path=/xryqqk-69/ 2024-04-09T23:48:04.120 DEBUG:boto:auth_path=/xryqqk-69/ 2024-04-09T23:48:04.120 DEBUG:boto:Method: HEAD 2024-04-09T23:48:04.120 DEBUG:boto:Path: /xryqqk-69/ 2024-04-09T23:48:04.120 DEBUG:boto:Data: 2024-04-09T23:48:04.121 DEBUG:boto:Headers: {} 2024-04-09T23:48:04.121 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.121 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.121 DEBUG:boto:Params: {} 2024-04-09T23:48:04.121 DEBUG:boto:Token: None 2024-04-09T23:48:04.121 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-69/ 2024-04-09T23:48:04.121 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo= 2024-04-09T23:48:04.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo=', 'Content-Length': '0'} 2024-04-09T23:48:04.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa32332e82b769b6-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.125 DEBUG:boto:path=/xryqqk-69/obj2 2024-04-09T23:48:04.125 DEBUG:boto:auth_path=/xryqqk-69/obj2 2024-04-09T23:48:04.125 DEBUG:boto:Method: PUT 2024-04-09T23:48:04.125 DEBUG:boto:Path: /xryqqk-69/obj2 2024-04-09T23:48:04.125 DEBUG:boto:Data: 2024-04-09T23:48:04.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:48:04.125 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.125 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.125 DEBUG:boto:Params: {} 2024-04-09T23:48:04.126 DEBUG:boto:Token: None 2024-04-09T23:48:04.126 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-69/obj2 2024-04-09T23:48:04.126 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:rNCuhyNB0ER1wAieZsj4aIwCLzU= 2024-04-09T23:48:04.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rNCuhyNB0ER1wAieZsj4aIwCLzU='} 2024-04-09T23:48:04.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b3c923e7e58f136-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.171 DEBUG:boto:path=/xryqqk-69/ 2024-04-09T23:48:04.171 DEBUG:boto:auth_path=/xryqqk-69/ 2024-04-09T23:48:04.171 DEBUG:boto:Method: HEAD 2024-04-09T23:48:04.171 DEBUG:boto:Path: /xryqqk-69/ 2024-04-09T23:48:04.171 DEBUG:boto:Data: 2024-04-09T23:48:04.171 DEBUG:boto:Headers: {} 2024-04-09T23:48:04.171 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.171 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.171 DEBUG:boto:Params: {} 2024-04-09T23:48:04.172 DEBUG:boto:Token: None 2024-04-09T23:48:04.172 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-69/ 2024-04-09T23:48:04.172 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo= 2024-04-09T23:48:04.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo=', 'Content-Length': '0'} 2024-04-09T23:48:04.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dd2ce4d7e81ab24-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.176 DEBUG:boto:path=/xryqqk-69/obj3 2024-04-09T23:48:04.176 DEBUG:boto:auth_path=/xryqqk-69/obj3 2024-04-09T23:48:04.176 DEBUG:boto:Method: PUT 2024-04-09T23:48:04.176 DEBUG:boto:Path: /xryqqk-69/obj3 2024-04-09T23:48:04.176 DEBUG:boto:Data: 2024-04-09T23:48:04.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:48:04.176 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.176 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.176 DEBUG:boto:Params: {} 2024-04-09T23:48:04.176 DEBUG:boto:Token: None 2024-04-09T23:48:04.177 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-69/obj3 2024-04-09T23:48:04.177 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:e7J09uuvnzMRs4FIlnS4S7Ny7NQ= 2024-04-09T23:48:04.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e7J09uuvnzMRs4FIlnS4S7Ny7NQ='} 2024-04-09T23:48:04.196 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000974e99a8755c0af3-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.196 DEBUG:boto:path=/xryqqk-70/ 2024-04-09T23:48:04.197 DEBUG:boto:auth_path=/xryqqk-70/ 2024-04-09T23:48:04.197 DEBUG:boto:Method: HEAD 2024-04-09T23:48:04.197 DEBUG:boto:Path: /xryqqk-70/ 2024-04-09T23:48:04.197 DEBUG:boto:Data: 2024-04-09T23:48:04.197 DEBUG:boto:Headers: {} 2024-04-09T23:48:04.197 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.197 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.197 DEBUG:boto:Params: {} 2024-04-09T23:48:04.197 DEBUG:boto:Token: None 2024-04-09T23:48:04.197 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-70/ 2024-04-09T23:48:04.198 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:QrAoEjzjDWjGE5wlXa05nSKuyyM= 2024-04-09T23:48:04.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QrAoEjzjDWjGE5wlXa05nSKuyyM=', 'Content-Length': '0'} 2024-04-09T23:48:04.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008854c339c8693280-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.211 DEBUG:boto:path=/xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-09T23:48:04.211 DEBUG:boto:auth_path=/xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-09T23:48:04.211 DEBUG:boto:Method: PUT 2024-04-09T23:48:04.211 DEBUG:boto:Path: /xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-09T23:48:04.211 DEBUG:boto:Data: 2024-04-09T23:48:04.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:48:04.212 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:48:04.212 DEBUG:boto:Port: 8000 2024-04-09T23:48:04.212 DEBUG:boto:Params: {} 2024-04-09T23:48:04.212 DEBUG:boto:Token: None 2024-04-09T23:48:04.212 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:48:04 GMT /xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-09T23:48:04.212 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AtR40feVszTUGUk4oXcu433avyY= 2024-04-09T23:48:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AtR40feVszTUGUk4oXcu433avyY='} 2024-04-09T23:48:04.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b22ba852d7c3e4f0-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:48:04.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.259 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:04.325 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:04.326 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:04.408 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706482.898476_1388.1", 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:02.898476Z" 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.465 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:48:04.466 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:04.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:48:04.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.467 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:04.539 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:04.539 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:04.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:04.633+0000 7f0f71e89780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:04.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:48:04.634+0000 7f0f71e89780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:48:04.667 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.668 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706482.898476_1388.1", 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:04.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:48:04.672 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:04Z" 2024-04-09T23:48:04.673 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:04.674 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706482.898476_1388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:02.898476Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:48:04Z" } 2024-04-09T23:48:04.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706482.898476_1388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:02.898476Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:48:04.675 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:04.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:48:04.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:48:04.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.675 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:04.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:04.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:04.854 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:48:04.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.854 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:04.854 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:04.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:48:04.855 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:48:04.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:04.856 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:48:04.857 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:04.857 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:48:04.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:04.857 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:04.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:04.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:05.006 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:05.006 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:05.006 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:05.007 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:05.008 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:05.009 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:05.010 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:05.011 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:05.012 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:05.012 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:05.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:05.012 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:04Z" 2024-04-09T23:48:05.012 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:05.013 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:04Z" } 2024-04-09T23:48:05.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:05.013 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:05.013 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:05.014 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:10.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:10.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:10.015 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:10.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:10.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:10.135 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:10.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:10.135 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:10.136 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:10.137 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:10.138 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:10.138 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:10.138 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:10.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:10.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:10.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:10.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:10.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:10Z" 2024-04-09T23:48:10.143 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:10.144 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:10Z" } 2024-04-09T23:48:10.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:10.145 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:10.145 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:10.145 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:15.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:15.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:15.147 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:15.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:15.206 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:15.280 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:15.280 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:15.280 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:15.280 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:15.280 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:15.280 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:15.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:15.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:15.283 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:15.284 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:15.285 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:15Z" 2024-04-09T23:48:15.286 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:15.287 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:15Z" } 2024-04-09T23:48:15.287 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:15.287 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:15.287 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:15.287 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:20.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:20.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:20.288 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:20.347 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:20.348 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:20.422 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:20.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:20.424 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:20.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:20.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:20.427 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:20Z" 2024-04-09T23:48:20.428 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:20.429 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:20Z" } 2024-04-09T23:48:20.429 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:20.429 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:20.430 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:20.430 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:25.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:25.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:25.431 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:25.490 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:25.491 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:25.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:25.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:25.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:25.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:25.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:25Z" 2024-04-09T23:48:25.576 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:25.577 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:25Z" } 2024-04-09T23:48:25.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:25.577 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:25.577 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:25.578 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:30.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:30.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:30.579 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:30.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:30.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:30.701 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:30Z" 2024-04-09T23:48:30.702 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:30.704 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:30Z" } 2024-04-09T23:48:30.704 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:30.704 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:30.704 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:30.704 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:35.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:35.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:35.706 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:35.768 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:35.768 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:35.846 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:35.846 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:35.847 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:35.848 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:35.849 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:35.850 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:35Z" 2024-04-09T23:48:35.851 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:35.852 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:35Z" } 2024-04-09T23:48:35.853 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:35.853 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:35.853 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:35.853 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:40.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:40.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:40.854 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:40.916 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:40.973 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:40.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:40.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:40.976 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:40.977 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:40.978 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:40.978 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:40.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:40.978 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:40Z" 2024-04-09T23:48:40.978 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:40.979 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:40Z" } 2024-04-09T23:48:40.979 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:40.979 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:40.979 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:40.979 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:45.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:45.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:45.981 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:46.041 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:46.041 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:46.115 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:46.116 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:46.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:46.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:46.119 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:46Z" 2024-04-09T23:48:46.120 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:46.121 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:46Z" } 2024-04-09T23:48:46.121 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:46.122 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:46.122 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:46.122 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:51.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:51.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:51.124 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:51.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:51.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:51.248 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:51.249 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:51.253 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:51.254 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:51.254 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:51.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:51.254 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:51Z" 2024-04-09T23:48:51.254 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:51.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:51Z" } 2024-04-09T23:48:51.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:51.255 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:51.255 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:51.255 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:48:56.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:56.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:48:56.257 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:48:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:48:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:48:56.399 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:48:56.399 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:48:56.399 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:48:56.399 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:56.400 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:48:56.401 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:56.402 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:48:56.403 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:48:56Z" 2024-04-09T23:48:56.404 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:48:56.406 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:48:56Z" } 2024-04-09T23:48:56.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:48:56.407 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:48:56.407 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:48:56.407 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:01.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:01.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:01.409 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:01.469 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:01.469 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:01.535 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:01.535 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:01.535 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:01.535 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:01.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:01.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:01.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:01.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:01Z" 2024-04-09T23:49:01.540 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:01.541 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:01Z" } 2024-04-09T23:49:01.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:01.542 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:01.542 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:01.542 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:06.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:06.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:06.544 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:06.604 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:06.604 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:06.661 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:06.661 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:06.662 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:06.663 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:06.664 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:06.665 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:06.666 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:06Z" 2024-04-09T23:49:06.667 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:06.670 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:06Z" } 2024-04-09T23:49:06.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:06.670 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:06.670 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:06.670 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:11.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:11.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:11.672 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:11.731 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:11.731 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:11.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:11.795 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:11.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:11Z" 2024-04-09T23:49:11.799 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:11.799 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:11Z" } 2024-04-09T23:49:11.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:11.800 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:11.800 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:11.800 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:16.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:16.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:16.801 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:16.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:16.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:16.929 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:16Z" 2024-04-09T23:49:16.935 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:16.936 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:16Z" } 2024-04-09T23:49:16.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:16.936 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:16.936 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:16.936 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:21.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:21.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:21.938 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:21.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:21.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:22.076 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:22Z" 2024-04-09T23:49:22.082 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:22.083 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:22Z" } 2024-04-09T23:49:22.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:22.083 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:22.084 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:22.084 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:27.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:27.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:27.085 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:27.145 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:27.145 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:27.221 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:27.222 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:27.223 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:27.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:27.225 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:27Z" 2024-04-09T23:49:27.226 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:27.227 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:27Z" } 2024-04-09T23:49:27.227 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:27.227 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:27.227 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:27.228 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:32.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:32.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:32.229 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:32.288 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:32.288 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:32.361 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:32.361 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:32.361 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:32.361 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:32.361 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:32.362 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:32.364 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:32.365 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:32Z" 2024-04-09T23:49:32.366 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:32.367 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:32Z" } 2024-04-09T23:49:32.368 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:32.368 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:32.368 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:32.368 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:37.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:37.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:37.370 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:37.429 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:37.430 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:37.499 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:37.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:37.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:37.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:37.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:37.504 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:37Z" 2024-04-09T23:49:37.505 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:37.506 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:37Z" } 2024-04-09T23:49:37.506 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:37.507 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:37.507 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:37.507 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:42.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:42.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:42.508 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:42.569 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:42.570 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:42.635 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:42.635 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:42.636 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:42.637 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:42.638 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:42.639 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:42Z" 2024-04-09T23:49:42.640 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:42.641 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:42Z" } 2024-04-09T23:49:42.642 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:42.642 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:42.642 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:42.642 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:47.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:47.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:47.644 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:47.778 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:47.779 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:47.780 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:47.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:47.782 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:47Z" 2024-04-09T23:49:47.783 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:47.784 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:47Z" } 2024-04-09T23:49:47.784 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:47.784 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:47.784 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:47.785 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:52.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:52.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:52.787 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:52.846 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:52.847 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:52.917 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:52.917 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:52.917 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:52.918 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:52.919 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:52.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:52.921 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:52Z" 2024-04-09T23:49:52.922 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:52.923 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:52Z" } 2024-04-09T23:49:52.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:52.923 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:52.923 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:52.924 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:49:57.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:57.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:49:57.924 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:49:57.984 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:49:57.984 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:49:58.046 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:49:58.046 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:49:58.047 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:58.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:58.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:49:58.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:49:58Z" 2024-04-09T23:49:58.051 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:49:58.052 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:49:58Z" } 2024-04-09T23:49:58.052 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:49:58.052 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:49:58.052 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:49:58.052 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:50:03.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:03.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:03.054 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:03.114 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:03.114 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:03.178 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:03.178 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:03.178 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:03.179 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000022969", 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:03.180 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019582", 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:03.181 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018339", 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:03.182 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:03Z" 2024-04-09T23:50:03.183 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:03.184 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022969", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:46:18.339197Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019582", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.108343Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018339", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:39:44.109973Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:03Z" } 2024-04-09T23:50:03.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-09T23:50:03.185 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:03.185 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-09T23:50:03.185 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-09T23:50:08.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.186 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:08.246 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:08.246 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:08.321 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:08.321 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:08.322 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:08.323 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.324 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.325 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.326 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:08Z" 2024-04-09T23:50:08.327 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:08.327 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:08Z" } 2024-04-09T23:50:08.328 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:08.328 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:08.328 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:08.328 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:50:08.328 DEBUG:boto:path=/xryqqk-69/ 2024-04-09T23:50:08.329 DEBUG:boto:auth_path=/xryqqk-69/ 2024-04-09T23:50:08.329 DEBUG:boto:Method: HEAD 2024-04-09T23:50:08.329 DEBUG:boto:Path: /xryqqk-69/ 2024-04-09T23:50:08.329 DEBUG:boto:Data: 2024-04-09T23:50:08.329 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.329 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:50:08.329 DEBUG:boto:Port: 8001 2024-04-09T23:50:08.329 DEBUG:boto:Params: {} 2024-04-09T23:50:08.330 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:50:08.330 DEBUG:boto:Token: None 2024-04-09T23:50:08.330 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-69/ 2024-04-09T23:50:08.330 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DrajcCcw39Gvn2PxDYGFFRPwiVc= 2024-04-09T23:50:08.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DrajcCcw39Gvn2PxDYGFFRPwiVc=', 'Content-Length': '0'} 2024-04-09T23:50:08.337 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f5f1be63cc77325-006615d430-24ae-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.337 DEBUG:boto:path=/xryqqk-69/obj1 2024-04-09T23:50:08.337 DEBUG:boto:auth_path=/xryqqk-69/obj1 2024-04-09T23:50:08.337 DEBUG:boto:Method: HEAD 2024-04-09T23:50:08.337 DEBUG:boto:Path: /xryqqk-69/obj1 2024-04-09T23:50:08.337 DEBUG:boto:Data: 2024-04-09T23:50:08.337 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.338 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:50:08.338 DEBUG:boto:Port: 8001 2024-04-09T23:50:08.338 DEBUG:boto:Params: {} 2024-04-09T23:50:08.338 DEBUG:boto:Token: None 2024-04-09T23:50:08.338 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-69/obj1 2024-04-09T23:50:08.338 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ArZzu2xy0zwuzcZpi15tiVYYL2E= 2024-04-09T23:50:08.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ArZzu2xy0zwuzcZpi15tiVYYL2E=', 'Content-Length': '0'} 2024-04-09T23:50:08.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af91cfa3dd69b2af-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.340 DEBUG:boto:path=/xryqqk-69/obj1 2024-04-09T23:50:08.340 DEBUG:boto:auth_path=/xryqqk-69/obj1 2024-04-09T23:50:08.341 DEBUG:boto:Method: GET 2024-04-09T23:50:08.341 DEBUG:boto:Path: /xryqqk-69/obj1 2024-04-09T23:50:08.341 DEBUG:boto:Data: 2024-04-09T23:50:08.341 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.341 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:50:08.341 DEBUG:boto:Port: 8001 2024-04-09T23:50:08.342 DEBUG:boto:Params: {} 2024-04-09T23:50:08.342 DEBUG:boto:Token: None 2024-04-09T23:50:08.342 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-69/obj1 2024-04-09T23:50:08.342 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:55ZH7HTcc6CEIVavEf7ELvrYG8w= 2024-04-09T23:50:08.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:55ZH7HTcc6CEIVavEf7ELvrYG8w=', 'Content-Length': '0'} 2024-04-09T23:50:08.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060cf0f9ff0dace25-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.344 DEBUG:boto:path=/xryqqk-69/obj2 2024-04-09T23:50:08.344 DEBUG:boto:auth_path=/xryqqk-69/obj2 2024-04-09T23:50:08.344 DEBUG:boto:Method: HEAD 2024-04-09T23:50:08.345 DEBUG:boto:Path: /xryqqk-69/obj2 2024-04-09T23:50:08.345 DEBUG:boto:Data: 2024-04-09T23:50:08.345 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.345 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:50:08.345 DEBUG:boto:Port: 8001 2024-04-09T23:50:08.345 DEBUG:boto:Params: {} 2024-04-09T23:50:08.345 DEBUG:boto:Token: None 2024-04-09T23:50:08.345 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-69/obj2 2024-04-09T23:50:08.345 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tlhL9IfhR+wfxCTdhsisAvnORJ4= 2024-04-09T23:50:08.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tlhL9IfhR+wfxCTdhsisAvnORJ4=', 'Content-Length': '0'} 2024-04-09T23:50:08.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a67d6e664fcc0ec-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.347 DEBUG:boto:path=/xryqqk-69/obj2 2024-04-09T23:50:08.347 DEBUG:boto:auth_path=/xryqqk-69/obj2 2024-04-09T23:50:08.347 DEBUG:boto:Method: GET 2024-04-09T23:50:08.347 DEBUG:boto:Path: /xryqqk-69/obj2 2024-04-09T23:50:08.347 DEBUG:boto:Data: 2024-04-09T23:50:08.347 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.348 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:50:08.348 DEBUG:boto:Port: 8001 2024-04-09T23:50:08.348 DEBUG:boto:Params: {} 2024-04-09T23:50:08.348 DEBUG:boto:Token: None 2024-04-09T23:50:08.348 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-69/obj2 2024-04-09T23:50:08.348 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cefVMWRO0rWiNUzZIDQZ6V9LmP8= 2024-04-09T23:50:08.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cefVMWRO0rWiNUzZIDQZ6V9LmP8=', 'Content-Length': '0'} 2024-04-09T23:50:08.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ec1751902d8577b-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.350 DEBUG:boto:path=/xryqqk-69/obj3 2024-04-09T23:50:08.350 DEBUG:boto:auth_path=/xryqqk-69/obj3 2024-04-09T23:50:08.350 DEBUG:boto:Method: HEAD 2024-04-09T23:50:08.350 DEBUG:boto:Path: /xryqqk-69/obj3 2024-04-09T23:50:08.350 DEBUG:boto:Data: 2024-04-09T23:50:08.351 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.351 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:50:08.351 DEBUG:boto:Port: 8001 2024-04-09T23:50:08.351 DEBUG:boto:Params: {} 2024-04-09T23:50:08.351 DEBUG:boto:Token: None 2024-04-09T23:50:08.351 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-69/obj3 2024-04-09T23:50:08.351 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:SNPiB75ri7QjEO/slJ8a5ylICc0= 2024-04-09T23:50:08.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SNPiB75ri7QjEO/slJ8a5ylICc0=', 'Content-Length': '0'} 2024-04-09T23:50:08.353 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008c47bec9c3609e2d-006615d430-24ae-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.355 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... FAIL 2024-04-09T23:50:08.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.357 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:08.405 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:08.405 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:08.442 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:50:08.461 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:50:08.461 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.461 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:08.461 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:50:08.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.462 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.462 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:08.462 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:50:08.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.463 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706482.898476_1388.1", 2024-04-09T23:50:08.463 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:02.898476Z" 2024-04-09T23:50:08.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.463 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.464 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:08.464 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:50:08.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.464 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:50:08.465 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:08.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:50:08.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.466 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:08.516 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:08.516 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:08.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:08.566+0000 7ff0d382b780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:08.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:08.567+0000 7ff0d382b780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:08.587 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:08.587 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.588 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.589 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706482.898476_1388.1", 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-09T23:50:08.591 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.592 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.593 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:08Z" 2024-04-09T23:50:08.594 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:08.595 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706482.898476_1388.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:02.898476Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:50:08Z" } 2024-04-09T23:50:08.595 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706482.898476_1388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:02.898476Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:50:08.595 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:08.595 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:08.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:50:08.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:08.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:08.596 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:08.641 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:08.698 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:50:08.699 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-71 2024-04-09T23:50:08.699 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:50:08.699 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:50:08.699 DEBUG:boto:Method: PUT 2024-04-09T23:50:08.699 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:50:08.699 DEBUG:boto:Data: 2024-04-09T23:50:08.699 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.699 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:50:08.699 DEBUG:boto:Port: 8000 2024-04-09T23:50:08.700 DEBUG:boto:Params: {} 2024-04-09T23:50:08.700 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:50:08.700 DEBUG:boto:Token: None 2024-04-09T23:50:08.700 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-71/ 2024-04-09T23:50:08.700 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Uz+KqslwzOUK8bb9JEF1z+f4YwI= 2024-04-09T23:50:08.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Uz+KqslwzOUK8bb9JEF1z+f4YwI=', 'Content-Length': '0'} 2024-04-09T23:50:08.728 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd861143639a5e32-006615d430-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.728 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-72 2024-04-09T23:50:08.728 DEBUG:boto:path=/xryqqk-72/ 2024-04-09T23:50:08.729 DEBUG:boto:auth_path=/xryqqk-72/ 2024-04-09T23:50:08.729 DEBUG:boto:Method: PUT 2024-04-09T23:50:08.729 DEBUG:boto:Path: /xryqqk-72/ 2024-04-09T23:50:08.729 DEBUG:boto:Data: 2024-04-09T23:50:08.729 DEBUG:boto:Headers: {} 2024-04-09T23:50:08.729 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:50:08.729 DEBUG:boto:Port: 8000 2024-04-09T23:50:08.729 DEBUG:boto:Params: {} 2024-04-09T23:50:08.729 DEBUG:boto:Token: None 2024-04-09T23:50:08.729 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:50:08 GMT /xryqqk-72/ 2024-04-09T23:50:08.730 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LjnXpAunFjnXPjB8msEM6w8yuGU= 2024-04-09T23:50:08.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LjnXpAunFjnXPjB8msEM6w8yuGU=', 'Content-Length': '0'} 2024-04-09T23:50:08.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dcd7a83cb1714ba6-006615d430-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:08.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.752 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:08.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:08.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:08.854 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:50:08.869 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:50:08.870 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:50:08.749146Z" 2024-04-09T23:50:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:50:08.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.871 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:50:08.872 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:08.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:50:08.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:08.873 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:08.924 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:08.924 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:08.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:08.958+0000 7fb5bbceb780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:08.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:08.960+0000 7fb5bbceb780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.981 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.744553_1408.1", 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.744553Z", 2024-04-09T23:50:08.982 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:08.983 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:08Z" 2024-04-09T23:50:08.984 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:08.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.744553_1408.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.744553Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:50:08Z" } 2024-04-09T23:50:08.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.744553_1408.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.744553Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:50:08.986 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:08.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.744553_1408.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:08.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706608.749146_1409.1 target=1_1712706608.744553_1408.1 2024-04-09T23:50:13.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:13.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:13.987 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:14.044 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.044 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:14.098+0000 7fb854412780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:14.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:14.099+0000 7fb854412780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:50:14.121 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:14.122 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:14.123 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:14.124 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.125 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:14.126 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:14Z" 2024-04-09T23:50:14.127 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:14.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:50:14Z" } 2024-04-09T23:50:14.129 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:50:14.129 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:14.129 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:14.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:50:14.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.130 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:14.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.181 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.240 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:50:14.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.240 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:14.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.347 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:50:14.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-71', '--dest-bucket=xryqqk-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-71', '--dest-bucket=xryqqk-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.348 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-71 --dest-bucket=xryqqk-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:14.402 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.402 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.471 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-71","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-72","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:50:14.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.472 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:14.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.583 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-71","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-72","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:50:14.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.584 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:14.631 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.632 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.692 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 14, 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:50:14.693 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:50:14.694 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:50:14.695 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:50:14.696 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:50:14.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:50:14.698 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-71", 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:50:14.700 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-72", 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.701 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "enabled" 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:50:14.702 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:50:14.703 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:50:14.704 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:50:14.705 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:50:14.706 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:50:14.707 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:50:14.708 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:50:14.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.709 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:50:14.709 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:50:14.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.709 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.716 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.717 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:50:14.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:50:14.719 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:14.720 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:14.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:50:14.722 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:50:14.770 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:14.770 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:14.855 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-71","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-72","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:50:14.855 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:50:14.855 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:50:14.855 DEBUG:boto:Method: HEAD 2024-04-09T23:50:14.856 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:50:14.856 DEBUG:boto:Data: 2024-04-09T23:50:14.856 DEBUG:boto:Headers: {} 2024-04-09T23:50:14.856 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:50:14.856 DEBUG:boto:Port: 8000 2024-04-09T23:50:14.856 DEBUG:boto:Params: {} 2024-04-09T23:50:14.856 DEBUG:boto:Token: None 2024-04-09T23:50:14.856 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:50:14 GMT /xryqqk-71/ 2024-04-09T23:50:14.856 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:z+OQu5rTti19ejDvHYpAMCy2q6Y= 2024-04-09T23:50:14.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:14 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:z+OQu5rTti19ejDvHYpAMCy2q6Y=', 'Content-Length': '0'} 2024-04-09T23:50:14.908 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000520ee4ddd76d1cf0-006615d436-25ea-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:14.909 DEBUG:boto:path=/xryqqk-71/obj1 2024-04-09T23:50:14.909 DEBUG:boto:auth_path=/xryqqk-71/obj1 2024-04-09T23:50:14.909 DEBUG:boto:Method: PUT 2024-04-09T23:50:14.909 DEBUG:boto:Path: /xryqqk-71/obj1 2024-04-09T23:50:14.909 DEBUG:boto:Data: 2024-04-09T23:50:14.910 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:50:14.910 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:50:14.910 DEBUG:boto:Port: 8000 2024-04-09T23:50:14.910 DEBUG:boto:Params: {} 2024-04-09T23:50:14.910 DEBUG:boto:Token: None 2024-04-09T23:50:14.910 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:50:14 GMT /xryqqk-71/obj1 2024-04-09T23:50:14.910 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VHBkV7ZhiN0SLDdo/WVRpMZwxGY= 2024-04-09T23:50:14.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:50:14 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VHBkV7ZhiN0SLDdo/WVRpMZwxGY='} 2024-04-09T23:50:14.939 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095ff9dc429a18db4-006615d436-25ea-a1'), ('Date', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:14.939 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:50:14.939 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:50:14.939 DEBUG:boto:Method: HEAD 2024-04-09T23:50:14.939 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:50:14.939 DEBUG:boto:Data: 2024-04-09T23:50:14.939 DEBUG:boto:Headers: {} 2024-04-09T23:50:14.939 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:50:14.939 DEBUG:boto:Port: 8000 2024-04-09T23:50:14.940 DEBUG:boto:Params: {} 2024-04-09T23:50:14.940 DEBUG:boto:Token: None 2024-04-09T23:50:14.940 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:50:14 GMT /xryqqk-71/ 2024-04-09T23:50:14.940 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:z+OQu5rTti19ejDvHYpAMCy2q6Y= 2024-04-09T23:50:14.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:14 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:z+OQu5rTti19ejDvHYpAMCy2q6Y=', 'Content-Length': '0'} 2024-04-09T23:50:14.942 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ede9305319263729-006615d436-25ea-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:14.942 DEBUG:boto:path=/xryqqk-71/obj2 2024-04-09T23:50:14.943 DEBUG:boto:auth_path=/xryqqk-71/obj2 2024-04-09T23:50:14.943 DEBUG:boto:Method: PUT 2024-04-09T23:50:14.943 DEBUG:boto:Path: /xryqqk-71/obj2 2024-04-09T23:50:14.943 DEBUG:boto:Data: 2024-04-09T23:50:14.943 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:50:14.943 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:50:14.943 DEBUG:boto:Port: 8000 2024-04-09T23:50:14.943 DEBUG:boto:Params: {} 2024-04-09T23:50:14.943 DEBUG:boto:Token: None 2024-04-09T23:50:14.943 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:50:14 GMT /xryqqk-71/obj2 2024-04-09T23:50:14.944 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zFx9b5zGeYC7MoaIHE2YeHYPnUg= 2024-04-09T23:50:14.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:50:14 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zFx9b5zGeYC7MoaIHE2YeHYPnUg='} 2024-04-09T23:50:14.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093aceb102c4785a4-006615d436-25ea-a1'), ('Date', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:50:14.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:14.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:14.987 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:15.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:15.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:15.168 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:50:15.233 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:50:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:50:15.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:50:08.749146Z" 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.234 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:50:15.235 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.235 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:50:15.236 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:15.236 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:50:15.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:15.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:15.236 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:15.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:15.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:15.389+0000 7f30bab16780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:15.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:50:15.391+0000 7f30bab16780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:50:15.444 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:50:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:50:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:50:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:50:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:15Z" 2024-04-09T23:50:15.450 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:15.451 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:50:15Z" } 2024-04-09T23:50:15.451 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:50:15.451 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:15.451 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:50:15.451 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:50:15.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:15.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:15.452 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:15.519 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:15.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:15.642 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023389", 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:50:14.937363Z" 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018621", 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:50:14.964067Z" 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.644 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:50:15.645 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:15.645 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:50:15.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:15.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:15.646 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:15.707 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:15.707 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:15.818 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:15Z" 2024-04-09T23:50:15.824 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:15.825 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:15Z" } 2024-04-09T23:50:15.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:15.825 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:15.825 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:15.825 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:20.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:20.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:20.826 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:20.884 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:20.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:20.959 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:20.959 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:20.959 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:20.960 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:20.961 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:20.962 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:20.963 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:20Z" 2024-04-09T23:50:20.964 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:20.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:20Z" } 2024-04-09T23:50:20.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:20.966 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:20.966 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:20.966 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:25.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:25.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:25.967 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:26.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:26.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:26.099 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:26.099 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:26.099 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:26.099 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:26.099 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:26.100 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:26.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:26.102 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:26.103 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:26Z" 2024-04-09T23:50:26.104 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:26.105 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:26Z" } 2024-04-09T23:50:26.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:26.105 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:26.106 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:26.106 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:31.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:31.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:31.108 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:31.183 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:31.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:31.263 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:31Z" 2024-04-09T23:50:31.269 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:31.270 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:31Z" } 2024-04-09T23:50:31.270 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:31.270 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:31.270 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:31.270 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:36.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:36.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:36.272 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:36.331 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:36.331 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:36.397 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:36.397 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:36.397 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:36.397 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:36.397 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:36.397 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:36.398 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:36.399 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:36.400 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:36.401 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:36.402 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:36.403 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:36.403 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:36.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:36.403 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:36Z" 2024-04-09T23:50:36.403 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:36.404 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:36Z" } 2024-04-09T23:50:36.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:36.404 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:36.404 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:36.405 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:41.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:41.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:41.406 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:41.464 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:41.464 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:41.534 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:41.534 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:41.534 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:41.534 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:41.534 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:41.534 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:41.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:41.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:41.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:41.538 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:41Z" 2024-04-09T23:50:41.539 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:41.540 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:41Z" } 2024-04-09T23:50:41.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:41.540 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:41.541 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:41.541 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:46.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:46.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:46.542 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:46.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:46.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:46.671 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:46.672 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:46.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:46.674 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:46.675 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:46.676 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:46.677 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:46.678 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:46Z" 2024-04-09T23:50:46.678 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:46.679 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:46Z" } 2024-04-09T23:50:46.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:46.679 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:46.679 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:46.680 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:51.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:51.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:51.681 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:51.740 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:51.741 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:51.808 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:51.809 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:51.810 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:51.811 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:51.812 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:51.813 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:51.813 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:51.813 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:51.813 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:51.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:51.814 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:51.814 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:51.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:51.814 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:51.814 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:51.815 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:51.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:51.815 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:51Z" 2024-04-09T23:50:51.815 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:51.816 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:51Z" } 2024-04-09T23:50:51.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:51.817 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:51.817 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:51.817 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:50:56.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:56.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:50:56.822 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:50:56.884 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:50:56.884 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:50:56.956 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:50:56.957 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:50:56.957 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:50:56.957 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:50:56.957 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:50:56.957 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:50:56.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:56.959 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:56.960 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:56.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:50:56.962 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:50:56.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:50:56.964 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:50:56Z" 2024-04-09T23:50:56.964 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:50:56.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:50:56Z" } 2024-04-09T23:50:56.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:50:56.965 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:50:56.965 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:50:56.965 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:01.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:01.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:01.968 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:02.027 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:02.104 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:02.105 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:02.106 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:02.108 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:02.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:02.110 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:02Z" 2024-04-09T23:51:02.110 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:02.110 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:02Z" } 2024-04-09T23:51:02.111 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:02.111 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:02.111 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:02.111 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:07.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:07.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:07.113 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:07.174 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:07.174 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:07.226 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:07.227 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:07.228 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:07.229 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:07.230 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:07.231 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:07Z" 2024-04-09T23:51:07.232 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:07.233 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:07Z" } 2024-04-09T23:51:07.233 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:07.233 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:07.233 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:07.233 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:12.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:12.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:12.235 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:12.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:12.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:12.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:12.374 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:12.375 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:12.376 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:12.377 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:12.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:12.378 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:12Z" 2024-04-09T23:51:12.378 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:12.379 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:12Z" } 2024-04-09T23:51:12.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:12.379 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:12.379 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:12.379 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:17.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:17.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:17.381 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:17.440 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:17.441 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:17.497 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:17.497 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:17Z" 2024-04-09T23:51:17.502 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:17.503 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:17Z" } 2024-04-09T23:51:17.503 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:17.504 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:17.504 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:17.504 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:22.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:22.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:22.505 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:22.564 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:22.564 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:22.644 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:22.645 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:22.646 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:22.647 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:22.648 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:22.649 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:22.650 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:22Z" 2024-04-09T23:51:22.650 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:22.651 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:22Z" } 2024-04-09T23:51:22.651 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:22.651 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:22.651 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:22.652 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:27.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:27.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:27.653 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:27.713 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:27.713 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:27.769 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:27.769 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:27.769 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:27.770 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:27.771 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:27.772 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:27.773 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:27.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:27.775 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:27Z" 2024-04-09T23:51:27.775 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:27.775 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:27Z" } 2024-04-09T23:51:27.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:27.776 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:27.776 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:27.776 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:32.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:32.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:32.778 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:32.837 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:32.837 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:32.910 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:32.910 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:32.910 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:32.910 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:32.911 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:32.912 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:32.913 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:32.914 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:32.915 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:32.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:32.916 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:32.916 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:32.916 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:32.916 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:32.916 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:32Z" 2024-04-09T23:51:32.917 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:32.917 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:32Z" } 2024-04-09T23:51:32.918 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:32.918 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:32.918 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:32.918 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:37.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:37.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:37.920 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:37.980 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:37.980 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:38.059 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:38.060 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:38.061 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:38.062 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:38.063 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:38.064 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:38.065 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:38.065 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:38.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:38.065 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:38Z" 2024-04-09T23:51:38.065 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:38.066 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:38Z" } 2024-04-09T23:51:38.066 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:38.066 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:38.066 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:38.066 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:43.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:43.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:43.068 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:43.126 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:43.126 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:43.198 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:43.199 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:43.200 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:43Z" 2024-04-09T23:51:43.204 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:43.205 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:43Z" } 2024-04-09T23:51:43.205 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:43.205 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:43.205 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:43.205 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:48.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:48.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:48.206 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:48.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:48.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:48.343 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:48.344 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:48.345 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:48.346 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:48.347 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:48.348 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:48Z" 2024-04-09T23:51:48.348 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:48.349 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:48Z" } 2024-04-09T23:51:48.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:48.349 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:48.349 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:48.349 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:53.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:53.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:53.351 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:53.410 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:53.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:53.490 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:53.491 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:53.492 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:53.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:53.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:53Z" 2024-04-09T23:51:53.495 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:53.496 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:53Z" } 2024-04-09T23:51:53.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:53.497 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:53.497 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:53.497 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:51:58.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:58.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:51:58.499 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:51:58.558 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:51:58.558 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:51:58.622 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:58.623 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:51:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:51:58.625 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:51:58.626 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:58.627 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:51:58.628 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:51:58.628 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:51:58.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:51:58.628 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:51:58Z" 2024-04-09T23:51:58.628 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:51:58.629 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:51:58Z" } 2024-04-09T23:51:58.629 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:51:58.629 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:51:58.629 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:51:58.629 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:52:03.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:03.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:03.631 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:03.690 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:03.691 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:03.754 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:03.754 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:03.755 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:03.756 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:03.757 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:03.758 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:03.759 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:52:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:52:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:03.760 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:03Z" 2024-04-09T23:52:03.760 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:03.761 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:52:03Z" } 2024-04-09T23:52:03.761 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:52:03.761 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:52:03.761 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:52:03.761 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:52:08.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:08.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:08.763 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:08.822 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:08.823 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:08.901 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:08.901 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:08.902 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:08.903 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:08.904 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:52:08.905 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:08Z" 2024-04-09T23:52:08.906 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:08.907 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:52:08Z" } 2024-04-09T23:52:08.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:52:08.907 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:52:08.907 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:52:08.907 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:52:13.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:13.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:13.908 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:13.969 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:13.969 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:14.028 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:14.028 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:14.028 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:14.028 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:14.028 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:52:14.029 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023249", 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:52:14.030 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:14.031 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018481", 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:14.032 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:52:14.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:14.034 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:14Z" 2024-04-09T23:52:14.034 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:14.035 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023249", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:48:04.194182Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018481", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.136792Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:52:14Z" } 2024-04-09T23:52:14.035 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-09T23:52:14.035 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:52:14.035 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-09T23:52:14.035 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023389 target=00000000000000000000:00000000000000023249, shard=3 master=00000000000000000000:00000000000000018621 target=00000000000000000000:00000000000000018481 2024-04-09T23:52:19.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:19.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:19.037 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:19.099 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.100 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:52:19.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:19.164 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023389", 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:52:19.165 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.937363Z" 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:19.166 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018621", 2024-04-09T23:52:19.167 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.964067Z" 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:52:19.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.169 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:19Z" 2024-04-09T23:52:19.169 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:19.170 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023389", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:50:14.937363Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:14.964067Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:52:19Z" } 2024-04-09T23:52:19.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023389', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.937363Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.964067Z'}}] 2024-04-09T23:52:19.170 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:52:19.170 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:52:19.170 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:52:19.171 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:52:19.171 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:52:19.171 DEBUG:boto:Method: HEAD 2024-04-09T23:52:19.171 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:52:19.171 DEBUG:boto:Data: 2024-04-09T23:52:19.171 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.171 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.171 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.171 DEBUG:boto:Params: {} 2024-04-09T23:52:19.172 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:52:19.172 DEBUG:boto:Token: None 2024-04-09T23:52:19.173 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-71/ 2024-04-09T23:52:19.173 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aafecnXWFrhzD0UiCieQqXWJfio= 2024-04-09T23:52:19.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aafecnXWFrhzD0UiCieQqXWJfio=', 'Content-Length': '0'} 2024-04-09T23:52:19.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aac2d80090dd6dfd-006615d4b3-25f6-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.179 DEBUG:boto:path=/xryqqk-71/obj1 2024-04-09T23:52:19.179 DEBUG:boto:auth_path=/xryqqk-71/obj1 2024-04-09T23:52:19.179 DEBUG:boto:Method: HEAD 2024-04-09T23:52:19.180 DEBUG:boto:Path: /xryqqk-71/obj1 2024-04-09T23:52:19.180 DEBUG:boto:Data: 2024-04-09T23:52:19.180 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.180 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.180 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.180 DEBUG:boto:Params: {} 2024-04-09T23:52:19.180 DEBUG:boto:Token: None 2024-04-09T23:52:19.180 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-71/obj1 2024-04-09T23:52:19.180 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:CfCh+BXi0nWcss9uWDnaAdv4Pos= 2024-04-09T23:52:19.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CfCh+BXi0nWcss9uWDnaAdv4Pos=', 'Content-Length': '0'} 2024-04-09T23:52:19.182 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000105f03645a33bafe-006615d4b3-25f6-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.182 DEBUG:boto:path=/xryqqk-71/obj2 2024-04-09T23:52:19.182 DEBUG:boto:auth_path=/xryqqk-71/obj2 2024-04-09T23:52:19.182 DEBUG:boto:Method: HEAD 2024-04-09T23:52:19.182 DEBUG:boto:Path: /xryqqk-71/obj2 2024-04-09T23:52:19.182 DEBUG:boto:Data: 2024-04-09T23:52:19.182 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.182 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.182 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.183 DEBUG:boto:Params: {} 2024-04-09T23:52:19.183 DEBUG:boto:Token: None 2024-04-09T23:52:19.183 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-71/obj2 2024-04-09T23:52:19.183 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:q//wpTGmPUI6hszznneUQUvNS4M= 2024-04-09T23:52:19.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:q//wpTGmPUI6hszznneUQUvNS4M=', 'Content-Length': '0'} 2024-04-09T23:52:19.184 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d1342d93ac6b0e36-006615d4b3-25f6-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.185 DEBUG:boto:path=/xryqqk-72/ 2024-04-09T23:52:19.185 DEBUG:boto:auth_path=/xryqqk-72/ 2024-04-09T23:52:19.185 DEBUG:boto:Method: HEAD 2024-04-09T23:52:19.185 DEBUG:boto:Path: /xryqqk-72/ 2024-04-09T23:52:19.185 DEBUG:boto:Data: 2024-04-09T23:52:19.185 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.185 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.185 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.185 DEBUG:boto:Params: {} 2024-04-09T23:52:19.185 DEBUG:boto:Token: None 2024-04-09T23:52:19.186 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-72/ 2024-04-09T23:52:19.186 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:8chcJmWETcm0oy8VFHuw3vG3M2o= 2024-04-09T23:52:19.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8chcJmWETcm0oy8VFHuw3vG3M2o=', 'Content-Length': '0'} 2024-04-09T23:52:19.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca189ea604502dc5-006615d4b3-25f6-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.188 DEBUG:boto:path=/xryqqk-72/obj1 2024-04-09T23:52:19.188 DEBUG:boto:auth_path=/xryqqk-72/obj1 2024-04-09T23:52:19.188 DEBUG:boto:Method: HEAD 2024-04-09T23:52:19.189 DEBUG:boto:Path: /xryqqk-72/obj1 2024-04-09T23:52:19.189 DEBUG:boto:Data: 2024-04-09T23:52:19.189 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.189 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.189 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.189 DEBUG:boto:Params: {} 2024-04-09T23:52:19.189 DEBUG:boto:Token: None 2024-04-09T23:52:19.189 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-72/obj1 2024-04-09T23:52:19.189 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:RqAeEaLoOTk3G16Z28Tq0Et8d/s= 2024-04-09T23:52:19.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RqAeEaLoOTk3G16Z28Tq0Et8d/s=', 'Content-Length': '0'} 2024-04-09T23:52:19.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2a7db98c9e82dcb-006615d4b3-25f6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.191 DEBUG:boto:path=/xryqqk-72/obj1 2024-04-09T23:52:19.191 DEBUG:boto:auth_path=/xryqqk-72/obj1 2024-04-09T23:52:19.191 DEBUG:boto:Method: GET 2024-04-09T23:52:19.192 DEBUG:boto:Path: /xryqqk-72/obj1 2024-04-09T23:52:19.192 DEBUG:boto:Data: 2024-04-09T23:52:19.192 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.192 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.192 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.192 DEBUG:boto:Params: {} 2024-04-09T23:52:19.192 DEBUG:boto:Token: None 2024-04-09T23:52:19.192 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-72/obj1 2024-04-09T23:52:19.192 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/MX9KCyZlIO7AMPJqdZpvWbKGCQ= 2024-04-09T23:52:19.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/MX9KCyZlIO7AMPJqdZpvWbKGCQ=', 'Content-Length': '0'} 2024-04-09T23:52:19.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086e5224d3ecba72a-006615d4b3-25f6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.194 DEBUG:boto:path=/xryqqk-72/obj2 2024-04-09T23:52:19.194 DEBUG:boto:auth_path=/xryqqk-72/obj2 2024-04-09T23:52:19.195 DEBUG:boto:Method: HEAD 2024-04-09T23:52:19.195 DEBUG:boto:Path: /xryqqk-72/obj2 2024-04-09T23:52:19.195 DEBUG:boto:Data: 2024-04-09T23:52:19.195 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.195 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.195 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.195 DEBUG:boto:Params: {} 2024-04-09T23:52:19.195 DEBUG:boto:Token: None 2024-04-09T23:52:19.195 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-72/obj2 2024-04-09T23:52:19.195 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Alc5aN0udlxtuXb53akzBO1BznE= 2024-04-09T23:52:19.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Alc5aN0udlxtuXb53akzBO1BznE=', 'Content-Length': '0'} 2024-04-09T23:52:19.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002d5a8d0a725c253-006615d4b3-25f6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.197 DEBUG:boto:path=/xryqqk-72/obj2 2024-04-09T23:52:19.197 DEBUG:boto:auth_path=/xryqqk-72/obj2 2024-04-09T23:52:19.197 DEBUG:boto:Method: GET 2024-04-09T23:52:19.197 DEBUG:boto:Path: /xryqqk-72/obj2 2024-04-09T23:52:19.198 DEBUG:boto:Data: 2024-04-09T23:52:19.198 DEBUG:boto:Headers: {} 2024-04-09T23:52:19.198 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:52:19.198 DEBUG:boto:Port: 8001 2024-04-09T23:52:19.198 DEBUG:boto:Params: {} 2024-04-09T23:52:19.198 DEBUG:boto:Token: None 2024-04-09T23:52:19.198 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:52:19 GMT /xryqqk-72/obj2 2024-04-09T23:52:19.198 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:UfSuf2dsM7pDib7Hla115GPLW1Q= 2024-04-09T23:52:19.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:52:19 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UfSuf2dsM7pDib7Hla115GPLW1Q=', 'Content-Length': '0'} 2024-04-09T23:52:19.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:50:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f03dae6b82193622-006615d4b3-25f6-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:52:19.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.201 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:19.251 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.252 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:19.319 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"enabled"}]} 2024-04-09T23:52:19.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.320 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:19.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:19.448 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:52:19.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.449 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:19.500 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.501 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:19.571 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:52:19.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.571 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:19.619 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.620 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:19.678 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:19.678 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:19.678 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 15, 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:52:19.679 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.680 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:52:19.681 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:52:19.682 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:52:19.683 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:52:19.684 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.685 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.687 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:52:19.688 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.689 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:52:19.689 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:52:19.689 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:52:19.689 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "allowed" 2024-04-09T23:52:19.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:52:19.691 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:52:19.692 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:52:19.693 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.693 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:52:19.693 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:52:19.693 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:52:19.693 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:52:19.693 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:52:19.694 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.698 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:52:19.698 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:52:19.698 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:52:19.698 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:52:19.699 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:52:19.700 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:52:19.701 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:52:19.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.704 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:52:19.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:52:19.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:52:19.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:52:19.708 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:19.709 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:19.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.709 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:19.762 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.762 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:19.860 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:52:19.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:19.860 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:19.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:19.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:20.101 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:52:20.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.102 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:20.177 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:20.177 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:20.324 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-09T23:52:20.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=xryqqk-71', '--source-bucket=*', '--dest-bucket=xryqqk-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=xryqqk-71', '--source-bucket=*', '--dest-bucket=xryqqk-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.325 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=xryqqk-71 '--source-bucket=*' --dest-bucket=xryqqk-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:20.391 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:20.392 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:20.519 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-72","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-09T23:52:20.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.520 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:20.699 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-72","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-09T23:52:20.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:52:20.699 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:52:20.751 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:20.752 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:20.860 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-72","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-09T23:52:20.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:20.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:20.860 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:20.914 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:20.914 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:20.969 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:20.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:52:20.652787Z" 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:20.985 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:52:20.986 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:20.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:52:20.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:20.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:20.987 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:21.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:21.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:21.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:21.076+0000 7f50cc53a780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:21.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:21.077+0000 7f50cc53a780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:21.097 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:21.097 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:21.097 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:21.098 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:21.099 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:21.100 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:21.101 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:21.102 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:21.103 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:21.103 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:21.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:21.103 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:21Z" 2024-04-09T23:52:21.103 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:21.104 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:21Z" } 2024-04-09T23:52:21.104 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:21.104 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:21.104 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:21.104 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:26.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:26.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:26.105 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:26.165 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:26.165 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:26.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:26.218+0000 7f8780953780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:26.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:26.219+0000 7f8780953780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:26.235 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:26.235 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:26.235 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:26.235 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:26.236 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:26.237 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:26.238 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:26.239 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:26.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:26.241 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:26.241 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:26.241 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:26.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:26.241 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:26Z" 2024-04-09T23:52:26.241 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:26.242 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:26Z" } 2024-04-09T23:52:26.242 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:26.242 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:26.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:26.242 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:31.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:31.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:31.244 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:31.301 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:31.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:31.339+0000 7f838cb71780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:31.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:31.340+0000 7f838cb71780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:31.355 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:31.356 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:31.357 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:31.358 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:31.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:31.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:31.361 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:31Z" 2024-04-09T23:52:31.362 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:31.362 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:31Z" } 2024-04-09T23:52:31.363 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:31.363 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:31.363 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:31.363 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:36.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:36.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:36.365 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:36.423 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:36.423 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:36.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:36.472+0000 7f224c811780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:36.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:36.474+0000 7f224c811780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:36.491 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:36.492 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:36.493 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:36.494 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:36.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:36.496 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:36Z" 2024-04-09T23:52:36.497 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:36.498 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:36Z" } 2024-04-09T23:52:36.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:36.499 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:36.499 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:36.499 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:41.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:41.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:41.501 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:41.558 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:41.559 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:41.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:41.598+0000 7f1ff9c58780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:41.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:41.599+0000 7f1ff9c58780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:41.614 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:41.614 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:41.614 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:41.614 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:41.615 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:41.616 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:41.617 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:41.618 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:41.619 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:41.620 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:41.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:41.620 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:41Z" 2024-04-09T23:52:41.620 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:41.621 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:41Z" } 2024-04-09T23:52:41.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:41.621 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:41.621 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:41.621 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:46.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:46.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:46.622 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:46.680 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:46.681 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:46.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:46.720+0000 7f39d4174780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:46.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:46.721+0000 7f39d4174780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:46.746 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:46.746 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:46.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:46.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:46.749 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:46.750 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:46.751 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:46.752 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:46.753 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:46Z" 2024-04-09T23:52:46.754 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:46.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:46Z" } 2024-04-09T23:52:46.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:46.755 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:46.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:46.755 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:51.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:51.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:51.757 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:51.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:51.817 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:51.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:51.862+0000 7fa565bc2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:51.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:51.863+0000 7fa565bc2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:51.880 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:51.880 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:51.880 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:51.881 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:51.882 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:51.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:51.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:51.885 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:51Z" 2024-04-09T23:52:51.886 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:51.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:51Z" } 2024-04-09T23:52:51.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:51.887 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:51.888 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:51.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:52:56.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:56.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:52:56.889 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:52:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:52:56.948 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:52:56.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:56.990+0000 7f38c4b28780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:56.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:52:56.992+0000 7f38c4b28780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:52:57.013 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:52:57.013 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:52:57.013 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:52:57.013 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:52:57.013 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:52:57.013 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:57.014 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:52:57.015 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:57.015 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:52:57.017 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:57.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:52:57.018 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:52:57.019 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:52:57.020 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:52:56Z" 2024-04-09T23:52:57.021 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:52:57.022 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:52:56Z" } 2024-04-09T23:52:57.023 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:52:57.023 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:57.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:52:57.023 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:02.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:02.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:02.025 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:02.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:02.137+0000 7f74ddcf9780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:02.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:02.138+0000 7f74ddcf9780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:02.159 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:02.160 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:02.161 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:02.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:02.163 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:02.164 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:02Z" 2024-04-09T23:53:02.165 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:02.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:02Z" } 2024-04-09T23:53:02.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:02.167 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:02.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:02.167 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:07.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:07.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:07.168 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:07.225 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:07.226 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:07.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:07.281+0000 7feb4122d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:07.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:07.282+0000 7feb4122d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:07.299 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:07.299 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:07.299 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:07.299 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:07.300 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:07.301 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:07.302 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:07.303 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:07.304 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:07.305 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:07.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:07.305 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:07Z" 2024-04-09T23:53:07.305 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:07.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:07Z" } 2024-04-09T23:53:07.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:07.306 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:07.306 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:07.306 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:12.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:12.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:12.308 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:12.366 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:12.366 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:12.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:12.403+0000 7f082a466780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:12.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:12.404+0000 7f082a466780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:12.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:12.420 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:12.421 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:12.422 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:12.423 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:12.424 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:12Z" 2024-04-09T23:53:12.425 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:12.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:12Z" } 2024-04-09T23:53:12.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:12.426 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:12.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:12.426 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:17.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:17.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:17.427 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:17.485 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:17.485 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:17.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:17.540+0000 7f7d660ad780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:17.541+0000 7f7d660ad780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:17Z" 2024-04-09T23:53:17.569 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:17.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:17Z" } 2024-04-09T23:53:17.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:17.570 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:17.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:17.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:22.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:22.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:22.571 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:22.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:22.676+0000 7fb73ccd3780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:22.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:22.680+0000 7fb73ccd3780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:22.700 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:22.700 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:22.700 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:22.701 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:22.702 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:22.703 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:22.704 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:22.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:22Z" 2024-04-09T23:53:22.706 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:22.707 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:22Z" } 2024-04-09T23:53:22.707 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:22.707 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:22.708 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:22.708 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:27.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:27.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:27.709 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:27.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:27.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:27.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:27.807+0000 7f5c287bb780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:27.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:27.808+0000 7f5c287bb780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:27.823 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:27.823 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:27.824 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:27.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:27.826 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:27.827 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:27.828 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:27.829 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:27Z" 2024-04-09T23:53:27.830 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:27.830 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:27Z" } 2024-04-09T23:53:27.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:27.831 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:27.831 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:27.831 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:32.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:32.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:32.833 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:32.893 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:32.894 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:32.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:32.941+0000 7fca07146780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:32.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:32.942+0000 7fca07146780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:32.960 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:32.961 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:32.962 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:32.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:32.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:32.965 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:32Z" 2024-04-09T23:53:32.966 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:32.967 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:32Z" } 2024-04-09T23:53:32.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:32.968 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:32.968 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:32.968 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:37.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:37.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:37.970 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:38.065+0000 7f84c5d2b780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:38.066+0000 7f84c5d2b780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:38.080 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:38.081 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:38.082 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:38.083 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:38.084 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:38.085 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:38.086 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:38.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:38.086 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:38Z" 2024-04-09T23:53:38.086 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:38.087 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:38Z" } 2024-04-09T23:53:38.087 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:38.087 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:38.087 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:38.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:43.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:43.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:43.088 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:43.146 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:43.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:43.186+0000 7f453c711780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:43.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:43.187+0000 7f453c711780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:43.201 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:43.201 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:43.202 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:43.203 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:43.204 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:43.205 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:43.206 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:43.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:43.207 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:43.207 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:43.207 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:43.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:43.208 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:43Z" 2024-04-09T23:53:43.208 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:43.209 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:43Z" } 2024-04-09T23:53:43.209 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:43.209 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:43.209 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:43.209 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:48.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:48.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:48.211 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:48.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:48.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:48.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:48.307+0000 7fb1ab20f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:48.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:48.308+0000 7fb1ab20f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:48.327 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:48.328 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:48.329 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:48.330 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:48.331 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:48.332 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:48Z" 2024-04-09T23:53:48.333 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:48.334 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:48Z" } 2024-04-09T23:53:48.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:48.335 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:48.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:48.335 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:53.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:53.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:53.338 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:53.396 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:53.396 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:53.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:53.437+0000 7f2a12bd0780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:53.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:53.438+0000 7f2a12bd0780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:53.455 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:53.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:53.457 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:53.458 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:53.459 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:53Z" 2024-04-09T23:53:53.460 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:53.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:53Z" } 2024-04-09T23:53:53.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:53.462 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:53.462 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:53.462 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:53:58.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:58.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:53:58.467 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:53:58.525 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:53:58.525 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:53:58.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:58.576+0000 7fd22240d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:58.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:53:58.577+0000 7fd22240d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:53:58.595 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:53:58.595 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:53:58.595 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:58.596 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:58.597 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:58.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:58.599 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:53:58.600 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:58.601 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:53:58.602 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:53:58.602 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:53:58.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:53:58.602 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:53:58Z" 2024-04-09T23:53:58.602 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:53:58.603 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:53:58Z" } 2024-04-09T23:53:58.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:53:58.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:58.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:53:58.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:54:03.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:03.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:03.605 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:03.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:03.713+0000 7f5362208780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:03.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:03.714+0000 7f5362208780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:03.733 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:03.733 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:03.734 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:03.735 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:03.736 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:03.737 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:03.738 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:03Z" 2024-04-09T23:54:03.739 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:03.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:03Z" } 2024-04-09T23:54:03.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:03.740 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:03.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:03.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:54:08.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:08.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:08.742 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:08.800 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:08.800 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:08.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:08.844+0000 7fd5ca03e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:08.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:08.845+0000 7fd5ca03e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:08.860 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:08.860 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:08.860 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:08.860 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:08.861 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:08.862 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:08.863 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:08.864 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:08.865 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:08Z" 2024-04-09T23:54:08.866 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:08.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:08Z" } 2024-04-09T23:54:08.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:08.869 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:08.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:08.869 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:54:13.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:13.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:13.871 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:13.929 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:13.929 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:13.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:13.981+0000 7f579f0a8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:13.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:13.982+0000 7f579f0a8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:14.004 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:14.004 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:14.004 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:14.005 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:14.006 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:14.007 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:14.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:14.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:14.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:14.008 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:14.008 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:14.008 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:14.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:14.010 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:14.010 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:14.010 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:14.010 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:14.010 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:14.010 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:14.011 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:13Z" 2024-04-09T23:54:14.012 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:14.014 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:13Z" } 2024-04-09T23:54:14.014 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:14.015 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:14.015 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:14.015 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:54:19.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:19.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:19.017 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:19.074 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:19.074 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:19.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:19.129+0000 7f0a16987780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:19.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:19.130+0000 7f0a16987780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:19.150 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:19.150 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:19.150 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:19.151 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:19.152 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:19.153 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706608.749146_1409.1", 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:19.154 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:08.749146Z", 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:19.155 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:19.157 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:19.158 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:19.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:19.159 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:19Z" 2024-04-09T23:54:19.159 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:19.160 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706608.749146_1409.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:08.749146Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:19Z" } 2024-04-09T23:54:19.160 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706608.749146_1409.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:08.749146Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:19.160 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:19.160 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706608.749146_1409.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:19.160 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706740.652787_1445.1 target=1_1712706608.749146_1409.1 2024-04-09T23:54:24.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.161 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:24.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:24.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:24.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:24.258+0000 7f6b96182780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:24.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:24.259+0000 7f6b96182780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:24.274 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.275 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:52:20.652787Z", 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.278 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:24.279 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:24Z" 2024-04-09T23:54:24.280 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:24.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706740.652787_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:52:20.652787Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:24Z" } 2024-04-09T23:54:24.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706740.652787_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:52:20.652787Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:24.282 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:24.282 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:24.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:54:24.282 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:54:24.282 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:54:24.282 DEBUG:boto:Method: HEAD 2024-04-09T23:54:24.283 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:54:24.283 DEBUG:boto:Data: 2024-04-09T23:54:24.283 DEBUG:boto:Headers: {} 2024-04-09T23:54:24.283 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:54:24.283 DEBUG:boto:Port: 8000 2024-04-09T23:54:24.283 DEBUG:boto:Params: {} 2024-04-09T23:54:24.283 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:54:24.284 DEBUG:boto:Token: None 2024-04-09T23:54:24.284 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:24 GMT /xryqqk-71/ 2024-04-09T23:54:24.284 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fkCEPj1KMXO6ocLcdhm3FfeeIN4= 2024-04-09T23:54:24.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fkCEPj1KMXO6ocLcdhm3FfeeIN4=', 'Content-Length': '0'} 2024-04-09T23:54:24.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009aeca7efe25ac5d4-006615d530-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:24.291 DEBUG:boto:path=/xryqqk-71/obj5 2024-04-09T23:54:24.291 DEBUG:boto:auth_path=/xryqqk-71/obj5 2024-04-09T23:54:24.291 DEBUG:boto:Method: PUT 2024-04-09T23:54:24.291 DEBUG:boto:Path: /xryqqk-71/obj5 2024-04-09T23:54:24.291 DEBUG:boto:Data: 2024-04-09T23:54:24.291 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:54:24.291 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:54:24.291 DEBUG:boto:Port: 8000 2024-04-09T23:54:24.291 DEBUG:boto:Params: {} 2024-04-09T23:54:24.291 DEBUG:boto:Token: None 2024-04-09T23:54:24.292 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:54:24 GMT /xryqqk-71/obj5 2024-04-09T23:54:24.292 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:YnaqElbHRuVd+3zlZF6e1PmjWok= 2024-04-09T23:54:24.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:54:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YnaqElbHRuVd+3zlZF6e1PmjWok='} 2024-04-09T23:54:24.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d15b0a7c9ab224b8-006615d530-2703-a1'), ('Date', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:24.298 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:54:24.298 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:54:24.299 DEBUG:boto:Method: HEAD 2024-04-09T23:54:24.299 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:54:24.299 DEBUG:boto:Data: 2024-04-09T23:54:24.299 DEBUG:boto:Headers: {} 2024-04-09T23:54:24.299 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:54:24.299 DEBUG:boto:Port: 8000 2024-04-09T23:54:24.299 DEBUG:boto:Params: {} 2024-04-09T23:54:24.299 DEBUG:boto:Token: None 2024-04-09T23:54:24.299 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:24 GMT /xryqqk-71/ 2024-04-09T23:54:24.299 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fkCEPj1KMXO6ocLcdhm3FfeeIN4= 2024-04-09T23:54:24.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fkCEPj1KMXO6ocLcdhm3FfeeIN4=', 'Content-Length': '0'} 2024-04-09T23:54:24.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b736da46078fb35a-006615d530-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:24.302 DEBUG:boto:path=/xryqqk-71/obj6 2024-04-09T23:54:24.302 DEBUG:boto:auth_path=/xryqqk-71/obj6 2024-04-09T23:54:24.302 DEBUG:boto:Method: PUT 2024-04-09T23:54:24.302 DEBUG:boto:Path: /xryqqk-71/obj6 2024-04-09T23:54:24.302 DEBUG:boto:Data: 2024-04-09T23:54:24.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:54:24.302 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:54:24.302 DEBUG:boto:Port: 8000 2024-04-09T23:54:24.302 DEBUG:boto:Params: {} 2024-04-09T23:54:24.302 DEBUG:boto:Token: None 2024-04-09T23:54:24.303 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:54:24 GMT /xryqqk-71/obj6 2024-04-09T23:54:24.303 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:w0kzLqCIe+22OUGq1h9DyHMFx0A= 2024-04-09T23:54:24.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:54:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w0kzLqCIe+22OUGq1h9DyHMFx0A='} 2024-04-09T23:54:24.308 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ecb6c2edf2bb099-006615d530-2703-a1'), ('Date', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:24.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.309 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:24.357 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:24.357 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:24.392 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:54:24.410 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:54:24.410 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:52:20.652787Z" 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.412 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:54:24.413 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:24.413 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:54:24.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.414 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:24.463 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:24.463 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:24.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:24.521+0000 7fc647aaa780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:24.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:24.522+0000 7fc647aaa780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:24.543 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:24.543 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:24.544 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.545 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.546 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:52:20.652787Z", 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.547 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:24.548 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.549 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:24Z" 2024-04-09T23:54:24.549 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:24.550 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706740.652787_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:52:20.652787Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:24Z" } 2024-04-09T23:54:24.550 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706740.652787_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:52:20.652787Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:24.550 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:24.550 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:24.550 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:54:24.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.551 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:24.600 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:24.600 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:24.663 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.665 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:54:24.666 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:24.666 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:54:24.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:24.667 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:24.718 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:24.718 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:54:24.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:54:24.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023389", 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.937363Z" 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018621", 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:24.790 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.964067Z" 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:24Z" 2024-04-09T23:54:24.791 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:24.792 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023389", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:50:14.937363Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:14.964067Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:54:24Z" } 2024-04-09T23:54:24.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023389', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.937363Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.964067Z'}}] 2024-04-09T23:54:24.793 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:24.793 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:54:24.793 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023529 target=00000000000000000000:00000000000000023389, shard=3 master=00000000000000000000:00000000000000018761 target=00000000000000000000:00000000000000018621 2024-04-09T23:54:29.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:29.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:29.795 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:29.852 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:29.852 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:29.921 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:29.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023389", 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.937363Z" 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:29.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:29.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018621", 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.964067Z" 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:29.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:54:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:54:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:29.926 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:29Z" 2024-04-09T23:54:29.926 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:29.927 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023389", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:50:14.937363Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:14.964067Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:54:29Z" } 2024-04-09T23:54:29.927 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023389', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.937363Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.964067Z'}}] 2024-04-09T23:54:29.927 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:29.927 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:54:29.930 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023529 target=00000000000000000000:00000000000000023389, shard=3 master=00000000000000000000:00000000000000018761 target=00000000000000000000:00000000000000018621 2024-04-09T23:54:34.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:34.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:34.932 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:34.991 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:34.992 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:35.047 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:35.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023389", 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.937363Z" 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:35.049 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:35.050 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018621", 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.964067Z" 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:35.051 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:35Z" 2024-04-09T23:54:35.052 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:35.053 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023389", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:50:14.937363Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:14.964067Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:54:35Z" } 2024-04-09T23:54:35.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023389', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.937363Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.964067Z'}}] 2024-04-09T23:54:35.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:35.054 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:54:35.054 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023529 target=00000000000000000000:00000000000000023389, shard=3 master=00000000000000000000:00000000000000018761 target=00000000000000000000:00000000000000018621 2024-04-09T23:54:40.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:40.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:40.055 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:40.115 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:40.116 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:40.179 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023389", 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:54:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.937363Z" 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018621", 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:50:14.964067Z" 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:40Z" 2024-04-09T23:54:40.184 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:40.185 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023389", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:50:14.937363Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018621", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:50:14.964067Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:54:40Z" } 2024-04-09T23:54:40.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023389', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.937363Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018621', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:50:14.964067Z'}}] 2024-04-09T23:54:40.186 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:40.186 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023389', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018621'} 2024-04-09T23:54:40.186 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023529 target=00000000000000000000:00000000000000023389, shard=3 master=00000000000000000000:00000000000000018761 target=00000000000000000000:00000000000000018621 2024-04-09T23:54:45.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:45.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:45.187 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:45.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:45.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:54:45.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011822", 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.309 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.310 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019723", 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.311 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.312 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.313 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:45Z" 2024-04-09T23:54:45.314 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:45.315 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011822", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:34:58.104853Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019723", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:48:04.241192Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:54:45Z" } 2024-04-09T23:54:45.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:54:45.315 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:45.315 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:54:45.315 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:54:45.315 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:54:45.316 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:54:45.316 DEBUG:boto:Method: HEAD 2024-04-09T23:54:45.316 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:54:45.316 DEBUG:boto:Data: 2024-04-09T23:54:45.316 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.316 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.316 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.316 DEBUG:boto:Params: {} 2024-04-09T23:54:45.317 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:54:45.317 DEBUG:boto:Token: None 2024-04-09T23:54:45.317 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-71/ 2024-04-09T23:54:45.317 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZESt08wRIBILs1m925A/E69tawA= 2024-04-09T23:54:45.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZESt08wRIBILs1m925A/E69tawA=', 'Content-Length': '0'} 2024-04-09T23:54:45.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009190e934bb3be92d-006615d545-26f9-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.326 DEBUG:boto:path=/xryqqk-71/obj5 2024-04-09T23:54:45.326 DEBUG:boto:auth_path=/xryqqk-71/obj5 2024-04-09T23:54:45.326 DEBUG:boto:Method: HEAD 2024-04-09T23:54:45.326 DEBUG:boto:Path: /xryqqk-71/obj5 2024-04-09T23:54:45.327 DEBUG:boto:Data: 2024-04-09T23:54:45.327 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.327 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.327 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.327 DEBUG:boto:Params: {} 2024-04-09T23:54:45.327 DEBUG:boto:Token: None 2024-04-09T23:54:45.327 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-71/obj5 2024-04-09T23:54:45.327 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Kj+ZmnyJ7tSmCy7X+mKBA/nNrb0= 2024-04-09T23:54:45.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kj+ZmnyJ7tSmCy7X+mKBA/nNrb0=', 'Content-Length': '0'} 2024-04-09T23:54:45.329 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000098ad41494a7fe3e5-006615d545-26f9-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.329 DEBUG:boto:path=/xryqqk-71/obj6 2024-04-09T23:54:45.329 DEBUG:boto:auth_path=/xryqqk-71/obj6 2024-04-09T23:54:45.329 DEBUG:boto:Method: HEAD 2024-04-09T23:54:45.329 DEBUG:boto:Path: /xryqqk-71/obj6 2024-04-09T23:54:45.329 DEBUG:boto:Data: 2024-04-09T23:54:45.330 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.330 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.330 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.330 DEBUG:boto:Params: {} 2024-04-09T23:54:45.330 DEBUG:boto:Token: None 2024-04-09T23:54:45.330 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-71/obj6 2024-04-09T23:54:45.330 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:p0vWTBRh3s00ytFXLV63QzmWL/c= 2024-04-09T23:54:45.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p0vWTBRh3s00ytFXLV63QzmWL/c=', 'Content-Length': '0'} 2024-04-09T23:54:45.332 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000000c48d771534da98f-006615d545-26f9-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.332 DEBUG:boto:path=/xryqqk-72/ 2024-04-09T23:54:45.332 DEBUG:boto:auth_path=/xryqqk-72/ 2024-04-09T23:54:45.332 DEBUG:boto:Method: HEAD 2024-04-09T23:54:45.332 DEBUG:boto:Path: /xryqqk-72/ 2024-04-09T23:54:45.332 DEBUG:boto:Data: 2024-04-09T23:54:45.332 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.332 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.332 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.332 DEBUG:boto:Params: {} 2024-04-09T23:54:45.333 DEBUG:boto:Token: None 2024-04-09T23:54:45.333 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-72/ 2024-04-09T23:54:45.333 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0VgX1QzjI4x2GX1DF55TkxKKMNM= 2024-04-09T23:54:45.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0VgX1QzjI4x2GX1DF55TkxKKMNM=', 'Content-Length': '0'} 2024-04-09T23:54:45.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000863a69bc31001991-006615d545-26f9-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.335 DEBUG:boto:path=/xryqqk-72/obj5 2024-04-09T23:54:45.335 DEBUG:boto:auth_path=/xryqqk-72/obj5 2024-04-09T23:54:45.335 DEBUG:boto:Method: HEAD 2024-04-09T23:54:45.335 DEBUG:boto:Path: /xryqqk-72/obj5 2024-04-09T23:54:45.335 DEBUG:boto:Data: 2024-04-09T23:54:45.335 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.336 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.336 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.336 DEBUG:boto:Params: {} 2024-04-09T23:54:45.336 DEBUG:boto:Token: None 2024-04-09T23:54:45.336 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-72/obj5 2024-04-09T23:54:45.336 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MQRYoAU8t3AKP0Q2u9OgTJVQI4E= 2024-04-09T23:54:45.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MQRYoAU8t3AKP0Q2u9OgTJVQI4E=', 'Content-Length': '0'} 2024-04-09T23:54:45.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096b32b133ed7d442-006615d545-26f9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.338 DEBUG:boto:path=/xryqqk-72/obj5 2024-04-09T23:54:45.338 DEBUG:boto:auth_path=/xryqqk-72/obj5 2024-04-09T23:54:45.339 DEBUG:boto:Method: GET 2024-04-09T23:54:45.339 DEBUG:boto:Path: /xryqqk-72/obj5 2024-04-09T23:54:45.339 DEBUG:boto:Data: 2024-04-09T23:54:45.339 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.339 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.339 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.339 DEBUG:boto:Params: {} 2024-04-09T23:54:45.339 DEBUG:boto:Token: None 2024-04-09T23:54:45.339 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-72/obj5 2024-04-09T23:54:45.339 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:mZs60EvSu2nG55fcSdQec1AEa0A= 2024-04-09T23:54:45.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mZs60EvSu2nG55fcSdQec1AEa0A=', 'Content-Length': '0'} 2024-04-09T23:54:45.341 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040c25c95ad908cbc-006615d545-26f9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.342 DEBUG:boto:path=/xryqqk-72/obj6 2024-04-09T23:54:45.342 DEBUG:boto:auth_path=/xryqqk-72/obj6 2024-04-09T23:54:45.342 DEBUG:boto:Method: HEAD 2024-04-09T23:54:45.342 DEBUG:boto:Path: /xryqqk-72/obj6 2024-04-09T23:54:45.342 DEBUG:boto:Data: 2024-04-09T23:54:45.342 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.342 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.342 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.342 DEBUG:boto:Params: {} 2024-04-09T23:54:45.343 DEBUG:boto:Token: None 2024-04-09T23:54:45.343 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-72/obj6 2024-04-09T23:54:45.343 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:f/3wMdw2Ue6ccud8qgyNuxtneCY= 2024-04-09T23:54:45.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:f/3wMdw2Ue6ccud8qgyNuxtneCY=', 'Content-Length': '0'} 2024-04-09T23:54:45.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a62570da2f94e4a0-006615d545-26f9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.345 DEBUG:boto:path=/xryqqk-72/obj6 2024-04-09T23:54:45.345 DEBUG:boto:auth_path=/xryqqk-72/obj6 2024-04-09T23:54:45.345 DEBUG:boto:Method: GET 2024-04-09T23:54:45.345 DEBUG:boto:Path: /xryqqk-72/obj6 2024-04-09T23:54:45.345 DEBUG:boto:Data: 2024-04-09T23:54:45.345 DEBUG:boto:Headers: {} 2024-04-09T23:54:45.345 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:54:45.345 DEBUG:boto:Port: 8001 2024-04-09T23:54:45.345 DEBUG:boto:Params: {} 2024-04-09T23:54:45.345 DEBUG:boto:Token: None 2024-04-09T23:54:45.346 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:54:45 GMT /xryqqk-72/obj6 2024-04-09T23:54:45.346 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:4iUcmKsOzYf2SakTkjducSnb7ps= 2024-04-09T23:54:45.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:54:45 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4iUcmKsOzYf2SakTkjducSnb7ps=', 'Content-Length': '0'} 2024-04-09T23:54:45.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:54:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-71:a5723776-f142-428c-a6df-d7bf949257da.9386.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000807e8513e2adc17a-006615d545-26f9-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:54:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:54:45.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:54:45.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:54:45.348 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:54:45.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:45.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:45.499 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-09T23:54:45.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:45.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:45.502 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:45.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:45.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:45.597 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:54:45.612 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.613 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706885.470177_1457.1", 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:54:45.470177Z" 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.614 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:54:45.615 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:45.615 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:54:45.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:45.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:45.616 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:45.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:45.711+0000 7fbd22b8d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:45.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:45.712+0000 7fbd22b8d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:45.733 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:45.733 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:45.733 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.734 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:45.738 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.739 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.740 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:52:20.652787Z", 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:45.741 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:45.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:45.743 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:45Z" 2024-04-09T23:54:45.743 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:45.744 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706740.652787_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:52:20.652787Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:45Z" } 2024-04-09T23:54:45.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706740.652787_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:52:20.652787Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:45.744 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:45.744 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:45.744 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706885.470177_1457.1 target=1_1712706740.652787_1445.1 2024-04-09T23:54:50.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:50.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:50.746 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:50.804 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:50.804 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:50.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:50.858+0000 7f1c90c7c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:50.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:50.859+0000 7f1c90c7c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:50.878 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:50.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:50.880 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:50.881 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:50.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:50.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:50.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:50.882 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:50.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:50.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:52:20.652787Z", 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:50.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:50.884 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:50Z" 2024-04-09T23:54:50.885 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:50.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706740.652787_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:52:20.652787Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:50Z" } 2024-04-09T23:54:50.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706740.652787_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:52:20.652787Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:50.887 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:50.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:50.887 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706885.470177_1457.1 target=1_1712706740.652787_1445.1 2024-04-09T23:54:55.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:55.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:54:55.889 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:54:55.946 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:54:55.946 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:54:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:55.999+0000 7f9e2aeaf780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:54:56.000+0000 7f9e2aeaf780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:54:56.023 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:54:56.024 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:56.025 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:56.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706740.652787_1445.1", 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:52:20.652787Z", 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:56.027 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:54:56.028 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:54:56Z" 2024-04-09T23:54:56.029 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:54:56.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706740.652787_1445.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:52:20.652787Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:54:56Z" } 2024-04-09T23:54:56.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706740.652787_1445.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:52:20.652787Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:54:56.030 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:56.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706740.652787_1445.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:54:56.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706885.470177_1457.1 target=1_1712706740.652787_1445.1 2024-04-09T23:55:01.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:01.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:01.032 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:01.091 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:01.091 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:01.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:01.146+0000 7f1b37704780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:01.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:01.147+0000 7f1b37704780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:01.174 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:01.175 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:01.176 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:01.177 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706885.465621_1456.1", 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:45.465621Z", 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:01.178 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:01.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:01.180 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:01.180 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:01.180 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:01.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:01.180 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:01Z" 2024-04-09T23:55:01.180 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:01.180 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706885.465621_1456.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:45.465621Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:01Z" } 2024-04-09T23:55:01.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706885.465621_1456.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:45.465621Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:01.181 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:01.181 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.465621_1456.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:01.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706885.470177_1457.1 target=1_1712706885.465621_1456.1 2024-04-09T23:55:06.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:06.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:06.182 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:06.240 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.240 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:06.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:06.282+0000 7fa294d58780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:06.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:06.283+0000 7fa294d58780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:06.299 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:06.300 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:06.301 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:06.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706885.470177_1457.1", 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:45.470177Z", 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:06.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:06.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:06Z" 2024-04-09T23:55:06.305 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:06.306 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706885.470177_1457.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:45.470177Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:06Z" } 2024-04-09T23:55:06.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706885.470177_1457.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:45.470177Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:06.307 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:06.307 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:06.307 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:55:06.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.307 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:06.356 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.356 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:06.412 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[]} 2024-04-09T23:55:06.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.413 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:06.464 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.464 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:06.520 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:55:06.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.521 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:06.577 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.577 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:06.633 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-09T23:55:06.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=xryqqk-71', '--source-bucket=xryqqk-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=xryqqk-71', '--source-bucket=xryqqk-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.634 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=xryqqk-71 --source-bucket=xryqqk-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.693 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:06.790 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-72","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-09T23:55:06.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.791 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:06.842 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.842 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:06.915 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-72","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-09T23:55:06.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:06.916 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:06.963 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:06.963 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:07.013 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-72","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-09T23:55:07.014 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:55:07.014 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:55:07.014 DEBUG:boto:Method: HEAD 2024-04-09T23:55:07.014 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:55:07.014 DEBUG:boto:Data: 2024-04-09T23:55:07.015 DEBUG:boto:Headers: {} 2024-04-09T23:55:07.015 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.015 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.015 DEBUG:boto:Params: {} 2024-04-09T23:55:07.015 DEBUG:boto:Token: None 2024-04-09T23:55:07.015 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-71/ 2024-04-09T23:55:07.015 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b7Tyl3eKXyCxMyspaAo+0mrUwVM= 2024-04-09T23:55:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b7Tyl3eKXyCxMyspaAo+0mrUwVM=', 'Content-Length': '0'} 2024-04-09T23:55:07.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009535936c0bef45cf-006615d55b-26f9-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.018 DEBUG:boto:path=/xryqqk-71/obj7 2024-04-09T23:55:07.018 DEBUG:boto:auth_path=/xryqqk-71/obj7 2024-04-09T23:55:07.018 DEBUG:boto:Method: PUT 2024-04-09T23:55:07.018 DEBUG:boto:Path: /xryqqk-71/obj7 2024-04-09T23:55:07.019 DEBUG:boto:Data: 2024-04-09T23:55:07.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:07.019 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.019 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.019 DEBUG:boto:Params: {} 2024-04-09T23:55:07.019 DEBUG:boto:Token: None 2024-04-09T23:55:07.019 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-71/obj7 2024-04-09T23:55:07.019 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:AvVRo1Vd25UcDgTNv1zPmvkJMsY= 2024-04-09T23:55:07.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AvVRo1Vd25UcDgTNv1zPmvkJMsY='} 2024-04-09T23:55:07.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000270c9fc2bf676112-006615d55b-26f9-a2'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.026 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:55:07.026 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:55:07.026 DEBUG:boto:Method: HEAD 2024-04-09T23:55:07.026 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:55:07.026 DEBUG:boto:Data: 2024-04-09T23:55:07.026 DEBUG:boto:Headers: {} 2024-04-09T23:55:07.026 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.026 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.027 DEBUG:boto:Params: {} 2024-04-09T23:55:07.027 DEBUG:boto:Token: None 2024-04-09T23:55:07.027 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-71/ 2024-04-09T23:55:07.027 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:b7Tyl3eKXyCxMyspaAo+0mrUwVM= 2024-04-09T23:55:07.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b7Tyl3eKXyCxMyspaAo+0mrUwVM=', 'Content-Length': '0'} 2024-04-09T23:55:07.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082b506a26abb2270-006615d55b-26f9-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.029 DEBUG:boto:path=/xryqqk-71/obj8 2024-04-09T23:55:07.029 DEBUG:boto:auth_path=/xryqqk-71/obj8 2024-04-09T23:55:07.029 DEBUG:boto:Method: PUT 2024-04-09T23:55:07.030 DEBUG:boto:Path: /xryqqk-71/obj8 2024-04-09T23:55:07.030 DEBUG:boto:Data: 2024-04-09T23:55:07.030 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:07.030 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.030 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.030 DEBUG:boto:Params: {} 2024-04-09T23:55:07.030 DEBUG:boto:Token: None 2024-04-09T23:55:07.030 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-71/obj8 2024-04-09T23:55:07.030 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fvro5I/XPCmajXBHuDaBO3Arsbk= 2024-04-09T23:55:07.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fvro5I/XPCmajXBHuDaBO3Arsbk='} 2024-04-09T23:55:07.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2e224f98e4db502-006615d55b-26f9-a2'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.075 DEBUG:boto:path=/xryqqk-72/ 2024-04-09T23:55:07.075 DEBUG:boto:auth_path=/xryqqk-72/ 2024-04-09T23:55:07.075 DEBUG:boto:Method: HEAD 2024-04-09T23:55:07.075 DEBUG:boto:Path: /xryqqk-72/ 2024-04-09T23:55:07.075 DEBUG:boto:Data: 2024-04-09T23:55:07.075 DEBUG:boto:Headers: {} 2024-04-09T23:55:07.075 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.075 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.076 DEBUG:boto:Params: {} 2024-04-09T23:55:07.076 DEBUG:boto:Token: None 2024-04-09T23:55:07.076 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-72/ 2024-04-09T23:55:07.076 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fuzIBC7+zADjDQCm3TboEteRsfY= 2024-04-09T23:55:07.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fuzIBC7+zADjDQCm3TboEteRsfY=', 'Content-Length': '0'} 2024-04-09T23:55:07.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000024657f627589317-006615d55b-26f9-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.078 DEBUG:boto:path=/xryqqk-72/obj9 2024-04-09T23:55:07.078 DEBUG:boto:auth_path=/xryqqk-72/obj9 2024-04-09T23:55:07.079 DEBUG:boto:Method: PUT 2024-04-09T23:55:07.079 DEBUG:boto:Path: /xryqqk-72/obj9 2024-04-09T23:55:07.079 DEBUG:boto:Data: 2024-04-09T23:55:07.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:07.079 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.079 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.079 DEBUG:boto:Params: {} 2024-04-09T23:55:07.079 DEBUG:boto:Token: None 2024-04-09T23:55:07.079 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-72/obj9 2024-04-09T23:55:07.079 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y7CHaNxQUd5jcp4CjfFdgq7zRBU= 2024-04-09T23:55:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7CHaNxQUd5jcp4CjfFdgq7zRBU='} 2024-04-09T23:55:07.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d61c83707b83fc21-006615d55b-26f9-a2'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.085 DEBUG:boto:path=/xryqqk-72/ 2024-04-09T23:55:07.085 DEBUG:boto:auth_path=/xryqqk-72/ 2024-04-09T23:55:07.085 DEBUG:boto:Method: HEAD 2024-04-09T23:55:07.085 DEBUG:boto:Path: /xryqqk-72/ 2024-04-09T23:55:07.085 DEBUG:boto:Data: 2024-04-09T23:55:07.085 DEBUG:boto:Headers: {} 2024-04-09T23:55:07.085 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.086 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.086 DEBUG:boto:Params: {} 2024-04-09T23:55:07.086 DEBUG:boto:Token: None 2024-04-09T23:55:07.086 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-72/ 2024-04-09T23:55:07.086 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:fuzIBC7+zADjDQCm3TboEteRsfY= 2024-04-09T23:55:07.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fuzIBC7+zADjDQCm3TboEteRsfY=', 'Content-Length': '0'} 2024-04-09T23:55:07.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c04859661aac5f34-006615d55b-26f9-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.088 DEBUG:boto:path=/xryqqk-72/obj10 2024-04-09T23:55:07.088 DEBUG:boto:auth_path=/xryqqk-72/obj10 2024-04-09T23:55:07.089 DEBUG:boto:Method: PUT 2024-04-09T23:55:07.089 DEBUG:boto:Path: /xryqqk-72/obj10 2024-04-09T23:55:07.089 DEBUG:boto:Data: 2024-04-09T23:55:07.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:07.089 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:55:07.089 DEBUG:boto:Port: 8001 2024-04-09T23:55:07.089 DEBUG:boto:Params: {} 2024-04-09T23:55:07.089 DEBUG:boto:Token: None 2024-04-09T23:55:07.089 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:07 GMT /xryqqk-72/obj10 2024-04-09T23:55:07.089 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5NLvzT9vCNXEqi77va+sv+t1cfA= 2024-04-09T23:55:07.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:07 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5NLvzT9vCNXEqi77va+sv+t1cfA='} 2024-04-09T23:55:07.094 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d9ab3c750b72c894-006615d55b-26f9-a2'), ('Date', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:07.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:07.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:07.095 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:07.144 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:07.145 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:07.206 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706906.897379_1465.1", 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:06.897379Z" 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:07.233 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:55:07.234 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:07.235 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:55:07.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:07.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:07.235 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:07.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:07.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:07.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:07.324+0000 7f70d17a1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:07.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:07.325+0000 7f70d17a1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.343 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:07.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:07.345 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706885.470177_1457.1", 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:45.470177Z", 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:07.346 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:07.347 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:07Z" 2024-04-09T23:55:07.348 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:07.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706885.470177_1457.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:45.470177Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:07Z" } 2024-04-09T23:55:07.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706885.470177_1457.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:45.470177Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:07.350 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:07.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:07.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706906.897379_1465.1 target=1_1712706885.470177_1457.1 2024-04-09T23:55:12.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:12.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:12.352 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:12.410 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:12.410 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:12.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:12.456+0000 7f8e91ba1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:12.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:12.457+0000 7f8e91ba1780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:12.472 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:12.472 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:12.472 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:12.472 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:12.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:12.473 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:12.474 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:12.475 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706885.470177_1457.1", 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:45.470177Z", 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:12.476 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:12.477 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:12.478 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:12.478 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:12.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:12.478 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:12Z" 2024-04-09T23:55:12.478 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:12.479 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706885.470177_1457.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:45.470177Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:12Z" } 2024-04-09T23:55:12.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706885.470177_1457.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:45.470177Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:12.479 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:12.479 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:12.479 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706906.897379_1465.1 target=1_1712706885.470177_1457.1 2024-04-09T23:55:17.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:17.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:17.481 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:17.538 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:17.538 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:17.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:17.583+0000 7f2ada404780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:17.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:17.584+0000 7f2ada404780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:17.600 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706885.470177_1457.1", 2024-04-09T23:55:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:45.470177Z", 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:17Z" 2024-04-09T23:55:17.606 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:17.607 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706885.470177_1457.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:45.470177Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:17Z" } 2024-04-09T23:55:17.607 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706885.470177_1457.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:45.470177Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:17.607 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:17.607 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706885.470177_1457.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:17.607 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706906.897379_1465.1 target=1_1712706885.470177_1457.1 2024-04-09T23:55:22.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:22.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:22.609 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:22.666 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:22.666 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:22.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:22.719+0000 7fb8662de780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:22.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:22.720+0000 7fb8662de780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:22.735 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:22.736 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.737 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706906.897379_1465.1", 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.738 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.739 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:22Z" 2024-04-09T23:55:22.740 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:22.741 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706906.897379_1465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:06.897379Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:22Z" } 2024-04-09T23:55:22.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:22.742 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:22.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:22.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:55:22.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:22.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:22.742 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:22.792 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:22.792 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:22.855 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000003891", 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:07.095316Z" 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011216", 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:07.026624Z" 2024-04-09T23:55:22.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000008971", 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:07.085924Z" 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000006864", 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:54:41.163417Z" 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.857 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:55:22.859 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000003891', 1: '00000000000000000000:00000000000000011216', 2: '00000000000000000000:00000000000000008971', 3: '00000000000000000000:00000000000000006864'} 2024-04-09T23:55:22.859 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-09T23:55:22.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:22.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:22.859 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:22.911 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:22.912 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:22.967 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:22.967 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 1955642440523705218 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:22.968 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000003891", 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:07.095316Z" 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:22.969 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011216", 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:07.026624Z" 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:22.970 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000008971", 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:07.085924Z" 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000006864", 2024-04-09T23:55:22.971 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:41.163417Z" 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 0, 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 0 2024-04-09T23:55:22.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:22.973 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:22Z" 2024-04-09T23:55:22.973 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:22.974 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 1955642440523705218 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003891", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:07.095316Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011216", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:07.026624Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008971", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:07.085924Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006864", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:41.163417Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-09T23:55:22Z" } 2024-04-09T23:55:22.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003891', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:07.095316Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011216', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:07.026624Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008971', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:07.085924Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006864', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:41.163417Z'}}] 2024-04-09T23:55:22.974 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003891', 1: '00000000000000000000:00000000000000011216', 2: '00000000000000000000:00000000000000008971', 3: '00000000000000000000:00000000000000006864'} 2024-04-09T23:55:22.974 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003891', 1: '00000000000000000000:00000000000000011216', 2: '00000000000000000000:00000000000000008971', 3: '00000000000000000000:00000000000000006864'} 2024-04-09T23:55:22.974 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-09T23:55:22.974 DEBUG:boto:path=/xryqqk-71/ 2024-04-09T23:55:22.974 DEBUG:boto:auth_path=/xryqqk-71/ 2024-04-09T23:55:22.975 DEBUG:boto:Method: HEAD 2024-04-09T23:55:22.975 DEBUG:boto:Path: /xryqqk-71/ 2024-04-09T23:55:22.975 DEBUG:boto:Data: 2024-04-09T23:55:22.975 DEBUG:boto:Headers: {} 2024-04-09T23:55:22.975 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:22.975 DEBUG:boto:Port: 8000 2024-04-09T23:55:22.975 DEBUG:boto:Params: {} 2024-04-09T23:55:22.975 DEBUG:boto:Token: None 2024-04-09T23:55:22.976 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:22 GMT /xryqqk-71/ 2024-04-09T23:55:22.976 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:G47ysTSCmxe9tOEF2rnrKJz1IGU= 2024-04-09T23:55:22.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:G47ysTSCmxe9tOEF2rnrKJz1IGU=', 'Content-Length': '0'} 2024-04-09T23:55:22.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007adc165f23131c5f-006615d56a-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:22.979 DEBUG:boto:path=/xryqqk-71/obj7 2024-04-09T23:55:22.979 DEBUG:boto:auth_path=/xryqqk-71/obj7 2024-04-09T23:55:22.979 DEBUG:boto:Method: HEAD 2024-04-09T23:55:22.979 DEBUG:boto:Path: /xryqqk-71/obj7 2024-04-09T23:55:22.979 DEBUG:boto:Data: 2024-04-09T23:55:22.979 DEBUG:boto:Headers: {} 2024-04-09T23:55:22.979 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:22.979 DEBUG:boto:Port: 8000 2024-04-09T23:55:22.979 DEBUG:boto:Params: {} 2024-04-09T23:55:22.980 DEBUG:boto:Token: None 2024-04-09T23:55:22.980 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:22 GMT /xryqqk-71/obj7 2024-04-09T23:55:22.980 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:zmfIkj2sE0mAG4udJZ/9fjMB5H8= 2024-04-09T23:55:22.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zmfIkj2sE0mAG4udJZ/9fjMB5H8=', 'Content-Length': '0'} 2024-04-09T23:55:22.981 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005c1796d9b1048e13-006615d56a-2703-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:55:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:22.981 DEBUG:boto:path=/xryqqk-71/obj8 2024-04-09T23:55:22.981 DEBUG:boto:auth_path=/xryqqk-71/obj8 2024-04-09T23:55:22.982 DEBUG:boto:Method: HEAD 2024-04-09T23:55:22.982 DEBUG:boto:Path: /xryqqk-71/obj8 2024-04-09T23:55:22.982 DEBUG:boto:Data: 2024-04-09T23:55:22.982 DEBUG:boto:Headers: {} 2024-04-09T23:55:22.982 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:22.982 DEBUG:boto:Port: 8000 2024-04-09T23:55:22.982 DEBUG:boto:Params: {} 2024-04-09T23:55:22.982 DEBUG:boto:Token: None 2024-04-09T23:55:22.982 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:22 GMT /xryqqk-71/obj8 2024-04-09T23:55:22.982 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tmVUrmbTWnPyEftzUly1OOj/uCM= 2024-04-09T23:55:22.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tmVUrmbTWnPyEftzUly1OOj/uCM=', 'Content-Length': '0'} 2024-04-09T23:55:22.984 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008de7b32e95acf40a-006615d56a-2703-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:55:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:22.984 DEBUG:boto:path=/xryqqk-71/obj9 2024-04-09T23:55:22.984 DEBUG:boto:auth_path=/xryqqk-71/obj9 2024-04-09T23:55:22.984 DEBUG:boto:Method: HEAD 2024-04-09T23:55:22.984 DEBUG:boto:Path: /xryqqk-71/obj9 2024-04-09T23:55:22.984 DEBUG:boto:Data: 2024-04-09T23:55:22.984 DEBUG:boto:Headers: {} 2024-04-09T23:55:22.985 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:22.985 DEBUG:boto:Port: 8000 2024-04-09T23:55:22.985 DEBUG:boto:Params: {} 2024-04-09T23:55:22.985 DEBUG:boto:Token: None 2024-04-09T23:55:22.985 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:22 GMT /xryqqk-71/obj9 2024-04-09T23:55:22.985 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9cZovpk2PJzvnbCxVbaGGyPmhBk= 2024-04-09T23:55:22.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9cZovpk2PJzvnbCxVbaGGyPmhBk=', 'Content-Length': '0'} 2024-04-09T23:55:22.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-72:a5723776-f142-428c-a6df-d7bf949257da.9386.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000868b9c7f55728f42-006615d56a-2703-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:55:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:22.987 DEBUG:boto:path=/xryqqk-71/obj9 2024-04-09T23:55:22.988 DEBUG:boto:auth_path=/xryqqk-71/obj9 2024-04-09T23:55:22.988 DEBUG:boto:Method: GET 2024-04-09T23:55:22.988 DEBUG:boto:Path: /xryqqk-71/obj9 2024-04-09T23:55:22.988 DEBUG:boto:Data: 2024-04-09T23:55:22.988 DEBUG:boto:Headers: {} 2024-04-09T23:55:22.988 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:22.989 DEBUG:boto:Port: 8000 2024-04-09T23:55:22.990 DEBUG:boto:Params: {} 2024-04-09T23:55:22.990 DEBUG:boto:Token: None 2024-04-09T23:55:22.990 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:55:22 GMT /xryqqk-71/obj9 2024-04-09T23:55:22.990 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kKpjuJI1VoteLR8vwf84W4RTMDI= 2024-04-09T23:55:22.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:22 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kKpjuJI1VoteLR8vwf84W4RTMDI=', 'Content-Length': '0'} 2024-04-09T23:55:22.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-72:a5723776-f142-428c-a6df-d7bf949257da.9386.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061b368140c1ffe3d-006615d56a-2703-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:55:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:23.038 DEBUG:boto:path=/xryqqk-71/obj10 2024-04-09T23:55:23.039 DEBUG:boto:auth_path=/xryqqk-71/obj10 2024-04-09T23:55:23.039 DEBUG:boto:Method: HEAD 2024-04-09T23:55:23.039 DEBUG:boto:Path: /xryqqk-71/obj10 2024-04-09T23:55:23.039 DEBUG:boto:Data: 2024-04-09T23:55:23.039 DEBUG:boto:Headers: {} 2024-04-09T23:55:23.039 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:23.039 DEBUG:boto:Port: 8000 2024-04-09T23:55:23.039 DEBUG:boto:Params: {} 2024-04-09T23:55:23.039 DEBUG:boto:Token: None 2024-04-09T23:55:23.040 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:23 GMT /xryqqk-71/obj10 2024-04-09T23:55:23.040 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:ZOUiihYs2zz+xhRDC9ryA1ybPCo= 2024-04-09T23:55:23.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZOUiihYs2zz+xhRDC9ryA1ybPCo=', 'Content-Length': '0'} 2024-04-09T23:55:23.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-72:a5723776-f142-428c-a6df-d7bf949257da.9386.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fad45c472cb7318-006615d56b-2703-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:55:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:23.042 DEBUG:boto:path=/xryqqk-71/obj10 2024-04-09T23:55:23.042 DEBUG:boto:auth_path=/xryqqk-71/obj10 2024-04-09T23:55:23.042 DEBUG:boto:Method: GET 2024-04-09T23:55:23.042 DEBUG:boto:Path: /xryqqk-71/obj10 2024-04-09T23:55:23.042 DEBUG:boto:Data: 2024-04-09T23:55:23.042 DEBUG:boto:Headers: {} 2024-04-09T23:55:23.042 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:23.043 DEBUG:boto:Port: 8000 2024-04-09T23:55:23.043 DEBUG:boto:Params: {} 2024-04-09T23:55:23.043 DEBUG:boto:Token: None 2024-04-09T23:55:23.043 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:55:23 GMT /xryqqk-71/obj10 2024-04-09T23:55:23.043 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:aXE9cAH1hf1zUmkk9Po7zosYfa4= 2024-04-09T23:55:23.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aXE9cAH1hf1zUmkk9Po7zosYfa4=', 'Content-Length': '0'} 2024-04-09T23:55:23.045 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'e208d06e-fe81-4216-93c9-2cb12ee81a65:xryqqk-72:a5723776-f142-428c-a6df-d7bf949257da.9386.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000421c0b2f75d92a1-006615d56b-2703-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:55:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:23.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:23.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:23.045 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket xryqqk-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:23.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:23.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:23.167 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-09T23:55:23.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:23.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:23.168 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:23.215 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:23.215 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:23.276 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-09T23:55:23.277 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2024-04-09T23:55:23.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:23.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:23.279 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:23.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:23.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:23.364 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:55:23.378 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:55:23.378 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706923.147638_1470.1", 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:23.147638Z" 2024-04-09T23:55:23.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.380 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.380 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:23.380 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:55:23.380 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:23.380 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:55:23.381 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:23.381 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:55:23.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:23.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:23.381 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:23.430 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:23.431 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:23.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:23.481+0000 7f1cf61c5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:23.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:23.482+0000 7f1cf61c5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:23.501 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706906.897379_1465.1", 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:23Z" 2024-04-09T23:55:23.507 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:23.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706906.897379_1465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:06.897379Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:23Z" } 2024-04-09T23:55:23.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:23.508 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:23.508 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:23.508 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-09T23:55:28.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:28.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:28.510 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:28.575 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:28.575 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:28.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:28.628+0000 7f3eca43c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:28.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:28.629+0000 7f3eca43c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:28.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:28.653 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:28.654 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706906.897379_1465.1", 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-09T23:55:28.655 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:28.656 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:28Z" 2024-04-09T23:55:28.657 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:28.658 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706906.897379_1465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:06.897379Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:28Z" } 2024-04-09T23:55:28.659 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:28.659 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:28.659 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:28.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-09T23:55:33.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:33.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:33.661 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:33.719 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:33.719 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:33.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:33.768+0000 7f5aef209780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:33.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:33.769+0000 7f5aef209780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:33.783 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:33.783 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:33.783 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:33.783 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:33.783 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:33.784 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:33.785 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:33.786 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706906.897379_1465.1", 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:33.787 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:33.788 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:33.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:33.789 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:33Z" 2024-04-09T23:55:33.789 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:33.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706906.897379_1465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:06.897379Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:33Z" } 2024-04-09T23:55:33.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:33.790 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:33.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:33.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-09T23:55:38.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:38.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:38.792 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:38.849 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:38.850 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:38.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:38.890+0000 7f6333081780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:38.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:38.891+0000 7f6333081780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:38.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:38.908 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:38.909 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706906.897379_1465.1", 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:38.910 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:38.911 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:38.912 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:38.912 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:38.912 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:38.912 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:38.912 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:38.912 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:38Z" 2024-04-09T23:55:38.913 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:38.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706906.897379_1465.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:06.897379Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:38Z" } 2024-04-09T23:55:38.914 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:38.914 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:38.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:38.915 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-09T23:55:43.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:43.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:43.916 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:43.975 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:43.975 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:44.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:44.018+0000 7ff9a8e3d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:44.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:44.020+0000 7ff9a8e3d780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:44.036 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:44.037 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.038 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.039 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706923.147638_1470.1", 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:23.147638Z", 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.040 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.041 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:44.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.042 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:44.042 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:44.042 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:44.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.043 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:44Z" 2024-04-09T23:55:44.043 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:44.044 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706923.147638_1470.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:23.147638Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:44Z" } 2024-04-09T23:55:44.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706923.147638_1470.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:23.147638Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:44.044 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:44.044 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:44.044 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:55:44.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:44.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:44.045 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:44.092 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:44.092 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:44.153 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[]} 2024-04-09T23:55:44.153 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-73 2024-04-09T23:55:44.154 DEBUG:boto:path=/xryqqk-73/ 2024-04-09T23:55:44.154 DEBUG:boto:auth_path=/xryqqk-73/ 2024-04-09T23:55:44.154 DEBUG:boto:Method: PUT 2024-04-09T23:55:44.154 DEBUG:boto:Path: /xryqqk-73/ 2024-04-09T23:55:44.154 DEBUG:boto:Data: 2024-04-09T23:55:44.154 DEBUG:boto:Headers: {} 2024-04-09T23:55:44.154 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:44.154 DEBUG:boto:Port: 8000 2024-04-09T23:55:44.154 DEBUG:boto:Params: {} 2024-04-09T23:55:44.155 DEBUG:boto:Token: None 2024-04-09T23:55:44.155 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:55:44 GMT /xryqqk-73/ 2024-04-09T23:55:44.155 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0mhVWn7hdvUDXXhhL9otRZbgOzg= 2024-04-09T23:55:44.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0mhVWn7hdvUDXXhhL9otRZbgOzg=', 'Content-Length': '0'} 2024-04-09T23:55:44.172 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d429d4233ba8e6e0-006615d580-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:44.172 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-74 2024-04-09T23:55:44.172 DEBUG:boto:path=/xryqqk-74/ 2024-04-09T23:55:44.172 DEBUG:boto:auth_path=/xryqqk-74/ 2024-04-09T23:55:44.173 DEBUG:boto:Method: PUT 2024-04-09T23:55:44.173 DEBUG:boto:Path: /xryqqk-74/ 2024-04-09T23:55:44.173 DEBUG:boto:Data: 2024-04-09T23:55:44.173 DEBUG:boto:Headers: {} 2024-04-09T23:55:44.173 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:44.173 DEBUG:boto:Port: 8000 2024-04-09T23:55:44.173 DEBUG:boto:Params: {} 2024-04-09T23:55:44.173 DEBUG:boto:Token: None 2024-04-09T23:55:44.173 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:55:44 GMT /xryqqk-74/ 2024-04-09T23:55:44.174 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:xjrJH59bibVvrOe0aI4iP3EEM3U= 2024-04-09T23:55:44.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xjrJH59bibVvrOe0aI4iP3EEM3U=', 'Content-Length': '0'} 2024-04-09T23:55:44.198 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a658ca698558098-006615d580-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:44.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:44.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:44.199 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:44.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:44.249 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:44.286 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:55:44.314 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.315 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:44.197906Z" 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.316 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:55:44.317 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:44.317 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:55:44.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:44.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:44.318 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:44.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:44.416+0000 7f4be70d9780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:44.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:44.416+0000 7f4be70d9780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:44.431 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:44.431 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:44.431 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:44.431 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:44.432 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:44.433 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.165143_1471.1", 2024-04-09T23:55:44.434 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.165143Z", 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:44.435 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:44.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.437 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:44.437 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:44.437 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:44.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:44.437 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:44Z" 2024-04-09T23:55:44.437 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:44.438 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.165143_1471.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.165143Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:44Z" } 2024-04-09T23:55:44.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.165143_1471.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.165143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:44.438 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:44.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.165143_1471.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:44.439 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712706944.197906_1478.1 target=1_1712706944.165143_1471.1 2024-04-09T23:55:49.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:49.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:49.440 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:49.550+0000 7f3a5a524780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:49.551+0000 7f3a5a524780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:49.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:49.570 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:49.571 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:55:49.572 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:49.573 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:49.574 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:49.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:49.575 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:49.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:49.575 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:49Z" 2024-04-09T23:55:49.575 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:49.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:49Z" } 2024-04-09T23:55:49.576 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:49.576 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:49.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:49.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:55:49.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.577 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:49.628 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:49.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:49.692 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:55:49.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.692 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:49.748 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:49.748 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:49.801 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2024-04-09T23:55:49.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-73', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-73', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.802 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xryqqk-73 --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-73 --dest-bucket=xryqqk-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:49.859 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:49.860 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:49.933 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:55:49.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-74', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-74', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:49.934 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xryqqk-74 --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-74 --dest-bucket=xryqqk-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:49.989 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:49.989 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:50.054 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:55:50.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:50.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:50.055 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:50.105 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:50.105 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:50.159 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:55:50.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:50.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:50.160 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:50.208 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:50.269 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 16, 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:55:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:55:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:55:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:55:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:55:50.274 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.275 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "directional": [ 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "source_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:55:50.276 INFO:teuthology.orchestra.run.smithi063.stdout: "dest_zone": "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-73", 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-73", 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:55:50.278 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-74", 2024-04-09T23:55:50.279 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:55:50.280 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:55:50.280 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:55:50.280 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:55:50.280 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.280 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "enabled" 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.282 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.283 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:55:50.284 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:55:50.286 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:55:50.287 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.288 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:55:50.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.290 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:50.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.292 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:55:50.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:55:50.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:50.295 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:50.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:50.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:55:50.296 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:55:50.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:50.350 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:50.439 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:55:50.439 DEBUG:boto:path=/xryqqk-73/ 2024-04-09T23:55:50.439 DEBUG:boto:auth_path=/xryqqk-73/ 2024-04-09T23:55:50.440 DEBUG:boto:Method: HEAD 2024-04-09T23:55:50.440 DEBUG:boto:Path: /xryqqk-73/ 2024-04-09T23:55:50.440 DEBUG:boto:Data: 2024-04-09T23:55:50.440 DEBUG:boto:Headers: {} 2024-04-09T23:55:50.440 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.440 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.440 DEBUG:boto:Params: {} 2024-04-09T23:55:50.440 DEBUG:boto:Token: None 2024-04-09T23:55:50.440 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-73/ 2024-04-09T23:55:50.441 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY= 2024-04-09T23:55:50.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY=', 'Content-Length': '0'} 2024-04-09T23:55:50.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d69de0c2d3d021f-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.451 DEBUG:boto:path=/xryqqk-73/obj1 2024-04-09T23:55:50.451 DEBUG:boto:auth_path=/xryqqk-73/obj1 2024-04-09T23:55:50.451 DEBUG:boto:Method: PUT 2024-04-09T23:55:50.451 DEBUG:boto:Path: /xryqqk-73/obj1 2024-04-09T23:55:50.452 DEBUG:boto:Data: 2024-04-09T23:55:50.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:50.452 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.452 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.452 DEBUG:boto:Params: {} 2024-04-09T23:55:50.452 DEBUG:boto:Token: None 2024-04-09T23:55:50.452 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-73/obj1 2024-04-09T23:55:50.452 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:z979YgZNneQ71Q5+Pv4xAOBz1SI= 2024-04-09T23:55:50.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:z979YgZNneQ71Q5+Pv4xAOBz1SI='} 2024-04-09T23:55:50.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4c2faa846a418ef-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.495 DEBUG:boto:path=/xryqqk-73/ 2024-04-09T23:55:50.495 DEBUG:boto:auth_path=/xryqqk-73/ 2024-04-09T23:55:50.495 DEBUG:boto:Method: HEAD 2024-04-09T23:55:50.495 DEBUG:boto:Path: /xryqqk-73/ 2024-04-09T23:55:50.495 DEBUG:boto:Data: 2024-04-09T23:55:50.495 DEBUG:boto:Headers: {} 2024-04-09T23:55:50.495 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.495 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.495 DEBUG:boto:Params: {} 2024-04-09T23:55:50.496 DEBUG:boto:Token: None 2024-04-09T23:55:50.496 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-73/ 2024-04-09T23:55:50.496 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY= 2024-04-09T23:55:50.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY=', 'Content-Length': '0'} 2024-04-09T23:55:50.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eaa4a83015bfa90-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.498 DEBUG:boto:path=/xryqqk-73/obj2 2024-04-09T23:55:50.499 DEBUG:boto:auth_path=/xryqqk-73/obj2 2024-04-09T23:55:50.499 DEBUG:boto:Method: PUT 2024-04-09T23:55:50.499 DEBUG:boto:Path: /xryqqk-73/obj2 2024-04-09T23:55:50.499 DEBUG:boto:Data: 2024-04-09T23:55:50.499 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:50.499 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.499 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.499 DEBUG:boto:Params: {} 2024-04-09T23:55:50.499 DEBUG:boto:Token: None 2024-04-09T23:55:50.500 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-73/obj2 2024-04-09T23:55:50.500 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yrZ2KNOCLAXghNpCGenzsZ07Th4= 2024-04-09T23:55:50.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yrZ2KNOCLAXghNpCGenzsZ07Th4='} 2024-04-09T23:55:50.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b37303bfba3f917-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.525 DEBUG:boto:path=/xryqqk-74/ 2024-04-09T23:55:50.525 DEBUG:boto:auth_path=/xryqqk-74/ 2024-04-09T23:55:50.525 DEBUG:boto:Method: HEAD 2024-04-09T23:55:50.525 DEBUG:boto:Path: /xryqqk-74/ 2024-04-09T23:55:50.525 DEBUG:boto:Data: 2024-04-09T23:55:50.525 DEBUG:boto:Headers: {} 2024-04-09T23:55:50.525 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.525 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.525 DEBUG:boto:Params: {} 2024-04-09T23:55:50.526 DEBUG:boto:Token: None 2024-04-09T23:55:50.526 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-74/ 2024-04-09T23:55:50.526 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8= 2024-04-09T23:55:50.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8=', 'Content-Length': '0'} 2024-04-09T23:55:50.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfb5c7328f7dd1a4-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.532 DEBUG:boto:path=/xryqqk-74/obj3 2024-04-09T23:55:50.532 DEBUG:boto:auth_path=/xryqqk-74/obj3 2024-04-09T23:55:50.533 DEBUG:boto:Method: PUT 2024-04-09T23:55:50.533 DEBUG:boto:Path: /xryqqk-74/obj3 2024-04-09T23:55:50.533 DEBUG:boto:Data: 2024-04-09T23:55:50.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:50.533 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.533 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.533 DEBUG:boto:Params: {} 2024-04-09T23:55:50.534 DEBUG:boto:Token: None 2024-04-09T23:55:50.534 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-74/obj3 2024-04-09T23:55:50.534 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:y03dgm/u/HHy2rfnJ9Ix+MJcLho= 2024-04-09T23:55:50.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y03dgm/u/HHy2rfnJ9Ix+MJcLho='} 2024-04-09T23:55:50.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010dc7e1db445dff8-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.583 DEBUG:boto:path=/xryqqk-74/ 2024-04-09T23:55:50.583 DEBUG:boto:auth_path=/xryqqk-74/ 2024-04-09T23:55:50.583 DEBUG:boto:Method: HEAD 2024-04-09T23:55:50.583 DEBUG:boto:Path: /xryqqk-74/ 2024-04-09T23:55:50.583 DEBUG:boto:Data: 2024-04-09T23:55:50.583 DEBUG:boto:Headers: {} 2024-04-09T23:55:50.583 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.583 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.583 DEBUG:boto:Params: {} 2024-04-09T23:55:50.584 DEBUG:boto:Token: None 2024-04-09T23:55:50.584 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-74/ 2024-04-09T23:55:50.584 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8= 2024-04-09T23:55:50.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8=', 'Content-Length': '0'} 2024-04-09T23:55:50.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1839a45a7fff274-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.587 DEBUG:boto:path=/xryqqk-74/obj4 2024-04-09T23:55:50.587 DEBUG:boto:auth_path=/xryqqk-74/obj4 2024-04-09T23:55:50.587 DEBUG:boto:Method: PUT 2024-04-09T23:55:50.587 DEBUG:boto:Path: /xryqqk-74/obj4 2024-04-09T23:55:50.587 DEBUG:boto:Data: 2024-04-09T23:55:50.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:55:50.587 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:55:50.587 DEBUG:boto:Port: 8000 2024-04-09T23:55:50.588 DEBUG:boto:Params: {} 2024-04-09T23:55:50.588 DEBUG:boto:Token: None 2024-04-09T23:55:50.588 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:55:50 GMT /xryqqk-74/obj4 2024-04-09T23:55:50.588 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:m6nz43gWEu9sNC+MLyAS13hA/Zg= 2024-04-09T23:55:50.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:m6nz43gWEu9sNC+MLyAS13hA/Zg='} 2024-04-09T23:55:50.609 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8992b5b1d79da06-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:55:50.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:50.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:50.609 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:50.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:50.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:50.796 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:55:50.853 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:55:50.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.853 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:50.853 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:55:50.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.853 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:44.197906Z" 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:50.854 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:55:50.855 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:50.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:55:50.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:50.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:50.856 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:50.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:50.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:51.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:51.008+0000 7feb6adc8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:51.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:55:51.009+0000 7feb6adc8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:55:51.053 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.054 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:51.055 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:51.056 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.057 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.058 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:51Z" 2024-04-09T23:55:51.059 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:51.060 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:55:51Z" } 2024-04-09T23:55:51.060 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:55:51.060 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:51.061 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:55:51.061 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:55:51.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:51.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:51.061 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:51.124 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:51.124 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:51.250 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:55:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:55:51.250 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:55:51.251 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:55:51.252 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.252 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:55:51.253 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:55:51.253 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:55:51.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:51.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:51.253 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:51.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:51.328 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:51.452 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:51.452 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:51.452 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:51.452 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:55:51.453 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:55:51.454 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.455 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:55:51.456 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:51Z" 2024-04-09T23:55:51.457 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:51.458 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:55:51Z" } 2024-04-09T23:55:51.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:55:51.458 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:55:51.459 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:55:51.459 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:55:56.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:56.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:55:56.460 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:55:56.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:55:56.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:55:56.583 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:55:56.584 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:56.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:55:56.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:55:56.587 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:55:56.588 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:56.589 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:55:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:55:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:55:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:55:56.590 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:55:56Z" 2024-04-09T23:55:56.590 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:55:56.591 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:55:56Z" } 2024-04-09T23:55:56.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:55:56.591 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:55:56.592 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:55:56.592 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:01.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:01.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:01.593 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:01.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:01.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:01.715 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:01.715 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:01.715 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:01.715 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:01.715 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:01.716 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:01.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:01.718 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:01.719 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:01.719 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:01.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:01.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:01.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:01.719 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:01.720 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:01.720 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:01.720 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:01.720 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:01.721 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:01.721 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:01.721 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:01.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:01.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:01.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:01.722 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:01.722 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:01.722 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:01.722 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:01.722 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:01.722 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:01.723 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:01.724 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:01.724 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:01.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:01.724 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:01Z" 2024-04-09T23:56:01.724 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:01.726 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:01Z" } 2024-04-09T23:56:01.726 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:01.726 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:01.726 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:01.726 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:06.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:06.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:06.728 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:06.788 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:06.789 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:06.855 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:06.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:06.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:06.858 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:06.859 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:06.860 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:06Z" 2024-04-09T23:56:06.861 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:06.862 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:06Z" } 2024-04-09T23:56:06.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:06.862 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:06.863 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:06.863 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:11.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:11.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:11.864 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:11.926 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:11.985 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:11.985 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:11.986 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:11.986 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:11.986 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:11.986 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:11.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:11.987 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:11.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:11.987 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:11.987 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:11.988 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:11.989 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:11.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:11.989 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:11.989 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:11.989 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:11.990 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:11.990 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:11.990 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:11.990 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:11.990 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:11.991 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:11.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:11.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:11.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:11.991 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:11.992 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:11.992 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:11.992 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:11.992 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:11.992 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:11.993 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:11.993 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:11.993 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:11.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:11.993 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:11.994 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:11.994 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:11.994 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:11.994 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:11.994 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:11.995 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:11.995 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:11.995 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:11.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:11.995 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:11.996 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:11.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:11.996 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:11.996 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:11.996 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:11.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:11.997 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:11Z" 2024-04-09T23:56:11.997 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:11.999 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:11Z" } 2024-04-09T23:56:11.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:12.000 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:12.000 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:12.000 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:17.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:17.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:17.002 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:17.062 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:17.062 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:17Z" 2024-04-09T23:56:17.122 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:17.123 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:17Z" } 2024-04-09T23:56:17.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:17.123 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:17.124 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:17.124 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:22.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:22.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:22.125 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:22.185 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:22.185 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:22.261 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:22.262 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:22.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:22.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:22.265 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:22Z" 2024-04-09T23:56:22.266 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:22.267 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:22Z" } 2024-04-09T23:56:22.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:22.267 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:22.267 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:22.268 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:27.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:27.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:27.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:27.330 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:27.330 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:27.387 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:27.387 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:27.388 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:27.389 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:27.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:27.391 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:27.392 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:27.393 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:27.394 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:27.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:27.396 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:27Z" 2024-04-09T23:56:27.397 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:27.398 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:27Z" } 2024-04-09T23:56:27.399 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:27.399 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:27.399 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:27.399 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:32.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:32.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:32.400 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:32.458 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:32.458 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:32.534 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:32.534 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:32.534 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:32.535 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:32.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:32.537 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:32.538 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:32.539 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:32.540 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:32Z" 2024-04-09T23:56:32.540 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:32.541 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:32Z" } 2024-04-09T23:56:32.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:32.542 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:32.542 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:32.542 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:37.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:37.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:37.544 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:37.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:37.604 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:37.666 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:37.666 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:37.666 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:37.666 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:37.667 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:37.668 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:37.669 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:37.670 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:37Z" 2024-04-09T23:56:37.671 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:37.672 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:37Z" } 2024-04-09T23:56:37.672 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:37.673 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:37.673 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:37.673 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:42.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:42.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:42.674 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:42.736 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:42.736 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:42.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:42.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:42.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:42.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:42Z" 2024-04-09T23:56:42.798 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:42.799 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:42Z" } 2024-04-09T23:56:42.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:42.799 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:42.799 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:42.800 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:47.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:47.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:47.801 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:47.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:47.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:47.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:47.932 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:47.933 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:47.934 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:47.935 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:47.936 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:47Z" 2024-04-09T23:56:47.937 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:47.939 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:47Z" } 2024-04-09T23:56:47.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:47.940 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:47.940 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:47.940 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:52.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:52.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:52.941 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:52.996 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:52.996 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:53.068 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:53.069 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:53.070 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:53.071 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:53.072 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:53.073 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:53.074 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:53Z" 2024-04-09T23:56:53.074 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:53.075 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:53Z" } 2024-04-09T23:56:53.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:53.075 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:53.075 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:53.076 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:56:58.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:58.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:56:58.077 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:56:58.138 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:56:58.138 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:56:58.208 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:56:58.208 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:58.209 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:56:58.210 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:58.211 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:56:58.212 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:58.213 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:56:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:56:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:56:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:56:58.214 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:56:58Z" 2024-04-09T23:56:58.214 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:56:58.215 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:56:58Z" } 2024-04-09T23:56:58.215 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:56:58.215 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:56:58.215 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:56:58.216 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:57:03.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:03.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:03.217 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:03.276 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:03.276 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:03.335 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:03.335 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:57:03.336 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:03.337 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:03.338 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:03.339 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:03Z" 2024-04-09T23:57:03.340 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:03.341 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:57:03Z" } 2024-04-09T23:57:03.342 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:57:03.342 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:03.342 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:57:03.342 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:57:08.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:08.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:08.344 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:08.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:08.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:08.480 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:08.480 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:08.481 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:08.482 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:08.483 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:08.484 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:57:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:08.485 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:57:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:08.486 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:57:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:57:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:08.487 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:08Z" 2024-04-09T23:57:08.487 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:08.488 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:57:08Z" } 2024-04-09T23:57:08.488 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:57:08.488 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:08.488 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:57:08.489 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:57:13.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:13.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:13.490 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:13.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:13.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:13.606 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:13.606 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:13.607 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:13.608 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:13.609 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:13.610 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:57:13.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:13.612 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:13Z" 2024-04-09T23:57:13.612 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:13.613 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:57:13Z" } 2024-04-09T23:57:13.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:57:13.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:13.613 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:57:13.613 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:57:18.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:18.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:18.615 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:18.674 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:18.675 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:18.743 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:18.743 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:18.743 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:18.743 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:18.743 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000011963", 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:18.744 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023529", 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:57:18.745 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:18.746 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018761", 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:18.747 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-09T23:57:18.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:18.748 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:18.748 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:18.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:18.748 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:18.748 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:57:18.749 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:57:18.749 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:18.749 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:18Z" 2024-04-09T23:57:18.749 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:18.750 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011963", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:21.137797Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023529", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:54:24.298968Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018761", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:54:24.309244Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:57:18Z" } 2024-04-09T23:57:18.750 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-09T23:57:18.750 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:18.750 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-09T23:57:18.750 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-09T23:57:23.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:23.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:23.752 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:23.814 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:23.814 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:23.868 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:23.870 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:23.871 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:23.872 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:57:23.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:23.874 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:23Z" 2024-04-09T23:57:23.874 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:23.875 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:57:23Z" } 2024-04-09T23:57:23.875 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:57:23.876 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:23.876 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:23.876 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:57:23.876 DEBUG:boto:path=/xryqqk-73/ 2024-04-09T23:57:23.876 DEBUG:boto:auth_path=/xryqqk-73/ 2024-04-09T23:57:23.876 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.876 DEBUG:boto:Path: /xryqqk-73/ 2024-04-09T23:57:23.876 DEBUG:boto:Data: 2024-04-09T23:57:23.876 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.877 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.877 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.877 DEBUG:boto:Params: {} 2024-04-09T23:57:23.877 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:57:23.877 DEBUG:boto:Token: None 2024-04-09T23:57:23.878 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-73/ 2024-04-09T23:57:23.878 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:APD2UEG4mEbmJjKRCnQebZI/puo= 2024-04-09T23:57:23.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:APD2UEG4mEbmJjKRCnQebZI/puo=', 'Content-Length': '0'} 2024-04-09T23:57:23.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049de10d8fcd13c17-006615d5e3-297c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.884 DEBUG:boto:path=/xryqqk-73/obj1 2024-04-09T23:57:23.884 DEBUG:boto:auth_path=/xryqqk-73/obj1 2024-04-09T23:57:23.884 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.884 DEBUG:boto:Path: /xryqqk-73/obj1 2024-04-09T23:57:23.884 DEBUG:boto:Data: 2024-04-09T23:57:23.884 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.884 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.885 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.885 DEBUG:boto:Params: {} 2024-04-09T23:57:23.885 DEBUG:boto:Token: None 2024-04-09T23:57:23.885 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-73/obj1 2024-04-09T23:57:23.885 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DfN/a70mmtdF5qmDet3SjjfHNBY= 2024-04-09T23:57:23.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DfN/a70mmtdF5qmDet3SjjfHNBY=', 'Content-Length': '0'} 2024-04-09T23:57:23.886 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c1f6c9035a35fb4f-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.887 DEBUG:boto:path=/xryqqk-73/obj2 2024-04-09T23:57:23.887 DEBUG:boto:auth_path=/xryqqk-73/obj2 2024-04-09T23:57:23.887 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.887 DEBUG:boto:Path: /xryqqk-73/obj2 2024-04-09T23:57:23.887 DEBUG:boto:Data: 2024-04-09T23:57:23.887 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.887 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.887 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.887 DEBUG:boto:Params: {} 2024-04-09T23:57:23.888 DEBUG:boto:Token: None 2024-04-09T23:57:23.888 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-73/obj2 2024-04-09T23:57:23.888 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kvodPwfiui4WB4EKH1HfEhksZaU= 2024-04-09T23:57:23.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kvodPwfiui4WB4EKH1HfEhksZaU=', 'Content-Length': '0'} 2024-04-09T23:57:23.889 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f602f698a10f200f-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.889 DEBUG:boto:path=/xryqqk-73/obj3 2024-04-09T23:57:23.889 DEBUG:boto:auth_path=/xryqqk-73/obj3 2024-04-09T23:57:23.890 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.890 DEBUG:boto:Path: /xryqqk-73/obj3 2024-04-09T23:57:23.890 DEBUG:boto:Data: 2024-04-09T23:57:23.890 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.890 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.890 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.890 DEBUG:boto:Params: {} 2024-04-09T23:57:23.890 DEBUG:boto:Token: None 2024-04-09T23:57:23.890 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-73/obj3 2024-04-09T23:57:23.890 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:wADqwrP4nEljfMG4nqwtq0gfqfQ= 2024-04-09T23:57:23.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wADqwrP4nEljfMG4nqwtq0gfqfQ=', 'Content-Length': '0'} 2024-04-09T23:57:23.892 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e3008a3664e55096-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.892 DEBUG:boto:path=/xryqqk-73/obj4 2024-04-09T23:57:23.892 DEBUG:boto:auth_path=/xryqqk-73/obj4 2024-04-09T23:57:23.892 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.892 DEBUG:boto:Path: /xryqqk-73/obj4 2024-04-09T23:57:23.892 DEBUG:boto:Data: 2024-04-09T23:57:23.892 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.892 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.893 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.893 DEBUG:boto:Params: {} 2024-04-09T23:57:23.893 DEBUG:boto:Token: None 2024-04-09T23:57:23.893 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-73/obj4 2024-04-09T23:57:23.893 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3mSdDhWVtJjJVAeVfw00MAqzBYw= 2024-04-09T23:57:23.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3mSdDhWVtJjJVAeVfw00MAqzBYw=', 'Content-Length': '0'} 2024-04-09T23:57:23.894 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000443530aaef48defd-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.895 DEBUG:boto:path=/xryqqk-74/ 2024-04-09T23:57:23.895 DEBUG:boto:auth_path=/xryqqk-74/ 2024-04-09T23:57:23.895 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.895 DEBUG:boto:Path: /xryqqk-74/ 2024-04-09T23:57:23.895 DEBUG:boto:Data: 2024-04-09T23:57:23.895 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.895 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.895 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.895 DEBUG:boto:Params: {} 2024-04-09T23:57:23.895 DEBUG:boto:Token: None 2024-04-09T23:57:23.896 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-74/ 2024-04-09T23:57:23.896 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:yl4idAmtktXujJFB+T2ZEl2xUbA= 2024-04-09T23:57:23.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yl4idAmtktXujJFB+T2ZEl2xUbA=', 'Content-Length': '0'} 2024-04-09T23:57:23.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ee5f2c3f9b3d02e-006615d5e3-297c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.898 DEBUG:boto:path=/xryqqk-74/obj1 2024-04-09T23:57:23.898 DEBUG:boto:auth_path=/xryqqk-74/obj1 2024-04-09T23:57:23.898 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.899 DEBUG:boto:Path: /xryqqk-74/obj1 2024-04-09T23:57:23.899 DEBUG:boto:Data: 2024-04-09T23:57:23.899 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.899 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.899 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.899 DEBUG:boto:Params: {} 2024-04-09T23:57:23.899 DEBUG:boto:Token: None 2024-04-09T23:57:23.899 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-74/obj1 2024-04-09T23:57:23.899 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:93eStWR0Gb/FoKMjK4zHXgcsl4c= 2024-04-09T23:57:23.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:93eStWR0Gb/FoKMjK4zHXgcsl4c=', 'Content-Length': '0'} 2024-04-09T23:57:23.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fa0981e72444877-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.901 DEBUG:boto:path=/xryqqk-74/obj1 2024-04-09T23:57:23.901 DEBUG:boto:auth_path=/xryqqk-74/obj1 2024-04-09T23:57:23.901 DEBUG:boto:Method: GET 2024-04-09T23:57:23.902 DEBUG:boto:Path: /xryqqk-74/obj1 2024-04-09T23:57:23.902 DEBUG:boto:Data: 2024-04-09T23:57:23.902 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.902 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.902 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.902 DEBUG:boto:Params: {} 2024-04-09T23:57:23.902 DEBUG:boto:Token: None 2024-04-09T23:57:23.902 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-74/obj1 2024-04-09T23:57:23.902 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:DJwM+OSEJPTqK4BAjy9kjpSIIdE= 2024-04-09T23:57:23.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DJwM+OSEJPTqK4BAjy9kjpSIIdE=', 'Content-Length': '0'} 2024-04-09T23:57:23.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f666c163ceb8655-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.946 DEBUG:boto:path=/xryqqk-74/obj2 2024-04-09T23:57:23.946 DEBUG:boto:auth_path=/xryqqk-74/obj2 2024-04-09T23:57:23.947 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.947 DEBUG:boto:Path: /xryqqk-74/obj2 2024-04-09T23:57:23.947 DEBUG:boto:Data: 2024-04-09T23:57:23.947 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.947 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.947 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.947 DEBUG:boto:Params: {} 2024-04-09T23:57:23.947 DEBUG:boto:Token: None 2024-04-09T23:57:23.947 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-74/obj2 2024-04-09T23:57:23.948 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Lejb2gMr09li79weeCr/v6jQzHw= 2024-04-09T23:57:23.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lejb2gMr09li79weeCr/v6jQzHw=', 'Content-Length': '0'} 2024-04-09T23:57:23.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8eb980382e78cf9-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.949 DEBUG:boto:path=/xryqqk-74/obj2 2024-04-09T23:57:23.950 DEBUG:boto:auth_path=/xryqqk-74/obj2 2024-04-09T23:57:23.950 DEBUG:boto:Method: GET 2024-04-09T23:57:23.950 DEBUG:boto:Path: /xryqqk-74/obj2 2024-04-09T23:57:23.950 DEBUG:boto:Data: 2024-04-09T23:57:23.950 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.950 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.950 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.950 DEBUG:boto:Params: {} 2024-04-09T23:57:23.950 DEBUG:boto:Token: None 2024-04-09T23:57:23.950 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-74/obj2 2024-04-09T23:57:23.951 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:vE9rIiYAdv9JECb3mtc4lyWnPBU= 2024-04-09T23:57:23.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vE9rIiYAdv9JECb3mtc4lyWnPBU=', 'Content-Length': '0'} 2024-04-09T23:57:23.952 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031712246af3eb1e3-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.953 DEBUG:boto:path=/xryqqk-74/obj3 2024-04-09T23:57:23.953 DEBUG:boto:auth_path=/xryqqk-74/obj3 2024-04-09T23:57:23.953 DEBUG:boto:Method: HEAD 2024-04-09T23:57:23.953 DEBUG:boto:Path: /xryqqk-74/obj3 2024-04-09T23:57:23.953 DEBUG:boto:Data: 2024-04-09T23:57:23.953 DEBUG:boto:Headers: {} 2024-04-09T23:57:23.953 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:57:23.953 DEBUG:boto:Port: 8001 2024-04-09T23:57:23.953 DEBUG:boto:Params: {} 2024-04-09T23:57:23.954 DEBUG:boto:Token: None 2024-04-09T23:57:23.954 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:23 GMT /xryqqk-74/obj3 2024-04-09T23:57:23.954 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OZCy6q1KIhQ4LdVnI547SVlvLaU= 2024-04-09T23:57:23.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OZCy6q1KIhQ4LdVnI547SVlvLaU=', 'Content-Length': '0'} 2024-04-09T23:57:23.955 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007503ac52b4400943-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:23.958 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... FAIL 2024-04-09T23:57:23.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:23.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:23.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:24.008 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:24.009 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:24.046 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:24.073 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:57:24.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.074 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:24.074 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:44.197906Z" 2024-04-09T23:57:24.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.074 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.075 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:24.075 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:57:24.075 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.075 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:57:24.076 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:24.077 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:57:24.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:24.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:24.077 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:24.128 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:24.129 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:24.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:24.181+0000 7f02b541e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:24.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:24.182+0000 7f02b541e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:24.201 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:24.201 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:24.201 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:24.201 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:24.201 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:24.201 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.202 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.203 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.204 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:24.205 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.206 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:24.206 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.206 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:24.206 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.206 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.206 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.209 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:24Z" 2024-04-09T23:57:24.210 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:24.211 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:24Z" } 2024-04-09T23:57:24.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:24.211 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:24.211 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:24.211 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:57:24.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:24.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:24.212 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:24.256 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:24.257 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:24.309 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:57:24.309 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-75 2024-04-09T23:57:24.309 DEBUG:boto:path=/xryqqk-75/ 2024-04-09T23:57:24.310 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-09T23:57:24.310 DEBUG:boto:Method: PUT 2024-04-09T23:57:24.310 DEBUG:boto:Path: /xryqqk-75/ 2024-04-09T23:57:24.310 DEBUG:boto:Data: 2024-04-09T23:57:24.310 DEBUG:boto:Headers: {} 2024-04-09T23:57:24.310 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:57:24.310 DEBUG:boto:Port: 8000 2024-04-09T23:57:24.310 DEBUG:boto:Params: {} 2024-04-09T23:57:24.310 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-09T23:57:24.311 DEBUG:boto:Token: None 2024-04-09T23:57:24.311 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:57:24 GMT /xryqqk-75/ 2024-04-09T23:57:24.311 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:K2tWsSJJJOP/AVjFxKRv7qxa3H0= 2024-04-09T23:57:24.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:K2tWsSJJJOP/AVjFxKRv7qxa3H0=', 'Content-Length': '0'} 2024-04-09T23:57:24.340 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d4ebe879c42b0c28-006615d5e4-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:24.340 INFO:rgw_multi.tests:create bucket zone=a1 name=xryqqk-76 2024-04-09T23:57:24.340 DEBUG:boto:path=/xryqqk-76/ 2024-04-09T23:57:24.340 DEBUG:boto:auth_path=/xryqqk-76/ 2024-04-09T23:57:24.341 DEBUG:boto:Method: PUT 2024-04-09T23:57:24.341 DEBUG:boto:Path: /xryqqk-76/ 2024-04-09T23:57:24.341 DEBUG:boto:Data: 2024-04-09T23:57:24.341 DEBUG:boto:Headers: {} 2024-04-09T23:57:24.341 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:57:24.341 DEBUG:boto:Port: 8000 2024-04-09T23:57:24.341 DEBUG:boto:Params: {} 2024-04-09T23:57:24.341 DEBUG:boto:Token: None 2024-04-09T23:57:24.341 DEBUG:boto:StringToSign: PUT Tue, 09 Apr 2024 23:57:24 GMT /xryqqk-76/ 2024-04-09T23:57:24.342 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:3D+9a362xFcmz00mcL+1kMyaHF0= 2024-04-09T23:57:24.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3D+9a362xFcmz00mcL+1kMyaHF0=', 'Content-Length': '0'} 2024-04-09T23:57:24.371 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fad228a10b8df4f3-006615d5e4-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:24.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:24.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:24.372 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:24.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:24.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:24.477 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:57:24.496 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.497 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:57:24.369051Z" 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.498 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:57:24.499 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:24.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:57:24.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:24.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:24.500 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:24.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:24.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:24.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:24.603+0000 7fd035e9f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:24.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:24.604+0000 7fd035e9f780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:24.626 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:24.626 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:24.627 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:24.627 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:24.627 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:24.627 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:24.627 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:24.628 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:24.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.630 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:24.631 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.632 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:24.633 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:24Z" 2024-04-09T23:57:24.633 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:24.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:24Z" } 2024-04-09T23:57:24.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:24.635 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:24.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:24.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707044.369051_1496.1 target=1_1712706944.197906_1478.1 2024-04-09T23:57:29.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:29.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:29.636 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:29.694 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:29.694 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:29.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:29.740+0000 7f330e2d5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:29.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:29.741+0000 7f330e2d5780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:29.756 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:29.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:29.758 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:29.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:29.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:29Z" 2024-04-09T23:57:29.761 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:29.762 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:29Z" } 2024-04-09T23:57:29.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:29.763 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:29.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:29.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707044.369051_1496.1 target=1_1712706944.197906_1478.1 2024-04-09T23:57:34.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:34.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:34.765 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:34.822 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:34.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:34.860+0000 7f8e29840780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:34.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:34.861+0000 7f8e29840780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:34.878 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:34.878 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:34.878 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:34.878 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:34.879 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:34.880 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:34.881 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:34.882 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:34.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:34.884 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:34.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:34.884 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:34Z" 2024-04-09T23:57:34.884 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:34.885 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:34Z" } 2024-04-09T23:57:34.885 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:34.885 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:34.885 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:34.885 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707044.369051_1496.1 target=1_1712706944.197906_1478.1 2024-04-09T23:57:39.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:39.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:39.887 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:39.945 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:39.945 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:39.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:39.995+0000 7f7d1c812780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:39.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:39.996+0000 7f7d1c812780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:40.011 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:40.012 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:40.013 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:40.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:40.015 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:40.016 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:39Z" 2024-04-09T23:57:40.017 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:40.018 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:39Z" } 2024-04-09T23:57:40.018 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:40.019 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:40.019 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:40.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707044.369051_1496.1 target=1_1712706944.197906_1478.1 2024-04-09T23:57:45.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:45.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:45.020 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:45.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:45.120+0000 7f4d01358780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:45.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:45.121+0000 7f4d01358780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:45.138 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:45.138 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:45.138 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:45.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:45.140 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:45.141 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:45.142 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:45.143 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:45.144 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:45.144 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:45.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:45.144 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:45Z" 2024-04-09T23:57:45.144 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:45.145 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:45Z" } 2024-04-09T23:57:45.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:45.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:45.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:45.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707044.369051_1496.1 target=1_1712706944.197906_1478.1 2024-04-09T23:57:50.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:50.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:50.147 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:50.204 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:50.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:50.252+0000 7f51bf0cc780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:50.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:50.253+0000 7f51bf0cc780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:50.268 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:50.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:50.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712706944.197906_1478.1", 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:50.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:50.272 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:50Z" 2024-04-09T23:57:50.273 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:50.274 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712706944.197906_1478.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:44.197906Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:50Z" } 2024-04-09T23:57:50.274 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:50.274 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:50.274 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:50.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707044.369051_1496.1 target=1_1712706944.197906_1478.1 2024-04-09T23:57:55.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:55.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:55.277 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:55.336 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:55.336 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:55.382+0000 7fb395323780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:55.383+0000 7fb395323780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:55.403 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:55.404 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:55.405 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:55.406 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:55.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:55.408 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:55Z" 2024-04-09T23:57:55.409 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:55.410 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:55Z" } 2024-04-09T23:57:55.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:55.410 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:55.411 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:55.411 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:57:55.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.411 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:55.461 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:55.462 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:55.518 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:57:55.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.519 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:55.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:55.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:55.640 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:57:55.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-75', '--dest-bucket=xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-75', '--dest-bucket=xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.641 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xryqqk-75 --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-75 --dest-bucket=xryqqk-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:55.695 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:55.695 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:55.774 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:57:55.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-75', '--dest-bucket=xryqqk-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-75', '--dest-bucket=xryqqk-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.775 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xryqqk-76 --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-75 --dest-bucket=xryqqk-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:55.835 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:55.835 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:55.919 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:57:55.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-75', '--dest-bucket=xryqqk-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-75', '--dest-bucket=xryqqk-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:55.919 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-75 --dest-bucket=xryqqk-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:55.974 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:55.974 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:56.025 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:57:56.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:56.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:56.026 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:56.076 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:56.076 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:56.150 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:57:56.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:56.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:56.151 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:56.200 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:56.200 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:56.289 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:56.289 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:56.289 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 17, 2024-04-09T23:57:56.289 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:57:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:57:56.293 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.294 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.295 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: "directional": [ 2024-04-09T23:57:56.296 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "source_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "dest_zone": "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-73", 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-73", 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.297 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.298 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-74", 2024-04-09T23:57:56.299 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.300 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-75", 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.302 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:57:56.303 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-76", 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:57:56.304 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-76", 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.305 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.306 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-76", 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:57:56.307 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "enabled" 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:57:56.309 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:57:56.310 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.312 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.312 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:57:56.312 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:57:56.312 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:57:56.312 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:57:56.313 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:57:56.314 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:57:56.315 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:57:56.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:57:56.317 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:57:56.318 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:57:56.319 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:57:56.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:56.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:56.321 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:57:56.321 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:57:56.321 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:56.321 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:56.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:56.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:57:56.322 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:57:56.368 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:56.368 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:56.457 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:57:56.458 DEBUG:boto:path=/xryqqk-75/ 2024-04-09T23:57:56.458 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-09T23:57:56.458 DEBUG:boto:Method: HEAD 2024-04-09T23:57:56.458 DEBUG:boto:Path: /xryqqk-75/ 2024-04-09T23:57:56.459 DEBUG:boto:Data: 2024-04-09T23:57:56.459 DEBUG:boto:Headers: {} 2024-04-09T23:57:56.459 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:57:56.459 DEBUG:boto:Port: 8000 2024-04-09T23:57:56.459 DEBUG:boto:Params: {} 2024-04-09T23:57:56.459 DEBUG:boto:Token: None 2024-04-09T23:57:56.459 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:56 GMT /xryqqk-75/ 2024-04-09T23:57:56.460 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Tsc9VCi2yEjOpjXIvlBc6Ug+7Wg= 2024-04-09T23:57:56.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:56 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Tsc9VCi2yEjOpjXIvlBc6Ug+7Wg=', 'Content-Length': '0'} 2024-04-09T23:57:56.765 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4800665104b44b6-006615d604-2acb-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:56.766 DEBUG:boto:path=/xryqqk-75/obj1 2024-04-09T23:57:56.766 DEBUG:boto:auth_path=/xryqqk-75/obj1 2024-04-09T23:57:56.766 DEBUG:boto:Method: PUT 2024-04-09T23:57:56.766 DEBUG:boto:Path: /xryqqk-75/obj1 2024-04-09T23:57:56.766 DEBUG:boto:Data: 2024-04-09T23:57:56.766 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:57:56.767 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:57:56.767 DEBUG:boto:Port: 8000 2024-04-09T23:57:56.767 DEBUG:boto:Params: {} 2024-04-09T23:57:56.767 DEBUG:boto:Token: None 2024-04-09T23:57:56.767 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:57:56 GMT /xryqqk-75/obj1 2024-04-09T23:57:56.767 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:PljIJTgPAXnLHc9caKVVfcgdeAQ= 2024-04-09T23:57:56.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:57:56 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PljIJTgPAXnLHc9caKVVfcgdeAQ='} 2024-04-09T23:57:56.807 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2430ea917e62003-006615d604-2acb-a1'), ('Date', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:56.807 DEBUG:boto:path=/xryqqk-75/ 2024-04-09T23:57:56.807 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-09T23:57:56.807 DEBUG:boto:Method: HEAD 2024-04-09T23:57:56.807 DEBUG:boto:Path: /xryqqk-75/ 2024-04-09T23:57:56.807 DEBUG:boto:Data: 2024-04-09T23:57:56.808 DEBUG:boto:Headers: {} 2024-04-09T23:57:56.808 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:57:56.808 DEBUG:boto:Port: 8000 2024-04-09T23:57:56.808 DEBUG:boto:Params: {} 2024-04-09T23:57:56.808 DEBUG:boto:Token: None 2024-04-09T23:57:56.808 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:57:56 GMT /xryqqk-75/ 2024-04-09T23:57:56.808 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Tsc9VCi2yEjOpjXIvlBc6Ug+7Wg= 2024-04-09T23:57:56.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:56 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Tsc9VCi2yEjOpjXIvlBc6Ug+7Wg=', 'Content-Length': '0'} 2024-04-09T23:57:56.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080df76fe9a7ef0cc-006615d604-2acb-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:56.811 DEBUG:boto:path=/xryqqk-75/obj2 2024-04-09T23:57:56.811 DEBUG:boto:auth_path=/xryqqk-75/obj2 2024-04-09T23:57:56.811 DEBUG:boto:Method: PUT 2024-04-09T23:57:56.811 DEBUG:boto:Path: /xryqqk-75/obj2 2024-04-09T23:57:56.811 DEBUG:boto:Data: 2024-04-09T23:57:56.811 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-09T23:57:56.812 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-09T23:57:56.812 DEBUG:boto:Port: 8000 2024-04-09T23:57:56.812 DEBUG:boto:Params: {} 2024-04-09T23:57:56.812 DEBUG:boto:Token: None 2024-04-09T23:57:56.812 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Tue, 09 Apr 2024 23:57:56 GMT /xryqqk-75/obj2 2024-04-09T23:57:56.812 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oNLuaHfawrhqQp8YwHKwlDo8Jts= 2024-04-09T23:57:56.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:57:56 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oNLuaHfawrhqQp8YwHKwlDo8Jts='} 2024-04-09T23:57:56.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000301c39cc59bd0f42-006615d604-2acb-a1'), ('Date', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:57:56.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:56.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:56.855 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:57.032 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:57:57.077 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:57:57.077 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.077 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:57:24.369051Z" 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.079 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:57.079 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:57:57.079 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.079 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:57:57.080 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:57.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:57:57.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:57.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:57.080 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:57.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:57.234+0000 7f844cc85780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:57.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:57:57.235+0000 7f844cc85780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:57:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:57.261 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.262 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:57.263 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.264 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.265 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.266 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:57Z" 2024-04-09T23:57:57.267 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:57.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:57:57Z" } 2024-04-09T23:57:57.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:57:57.268 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:57.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:57:57.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-09T23:57:57.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:57.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:57.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:57.321 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:57.321 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:57:56.805118Z" 2024-04-09T23:57:57.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:57:56.834128Z" 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.441 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:57:57.442 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:57:57.442 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:57:57.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:57.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:57:57.443 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:57:57.503 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:57:57.504 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:57:57.615 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.616 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.617 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:57.618 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:57:57.619 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:57:57.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:57:57.621 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:57:57Z" 2024-04-09T23:57:57.621 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:57:57.622 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:57:57Z" } 2024-04-09T23:57:57.622 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:57:57.622 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:57:57.622 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:57:57.622 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:02.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:02.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:02.624 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:02.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:02.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:02.742 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:02.742 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:02.742 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:02.742 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:02.742 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:02.743 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:02.744 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:02.745 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:02.746 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:02.747 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:02.748 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:02.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:02.748 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:02Z" 2024-04-09T23:58:02.748 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:02.749 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:02Z" } 2024-04-09T23:58:02.749 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:02.749 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:02.749 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:02.750 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:07.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:07.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:07.751 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:07.814 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:07.814 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:07.880 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:07.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:07.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:07.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:07.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:07.885 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:07Z" 2024-04-09T23:58:07.886 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:07.887 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:07Z" } 2024-04-09T23:58:07.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:07.888 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:07.888 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:07.888 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:12.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:12.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:12.890 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:12.948 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:12.948 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:13.003 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:13.004 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:13.005 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:13.006 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:13.007 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:13.008 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:13.009 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:13.009 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:13.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:13.009 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:12Z" 2024-04-09T23:58:13.009 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:13.010 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:12Z" } 2024-04-09T23:58:13.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:13.010 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:13.010 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:13.010 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:18.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:18.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:18.012 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:18.072 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:18.152 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:18.152 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:18.153 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:18.154 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:18.155 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:18.156 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:18Z" 2024-04-09T23:58:18.157 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:18.158 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:18Z" } 2024-04-09T23:58:18.158 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:18.159 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:18.159 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:18.159 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:23.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:23.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:23.160 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:23.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:23.221 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:23.283 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:23.284 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:23.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:23.286 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:23.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:23.288 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:23.288 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:23.288 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:23.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:23.288 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:23Z" 2024-04-09T23:58:23.288 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:23.289 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:23Z" } 2024-04-09T23:58:23.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:23.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:23.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:23.290 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:28.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:28.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:28.291 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:28.351 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:28.351 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:28.424 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:28.424 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:28.424 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:28.425 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:28.426 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:28.427 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:28.428 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:28Z" 2024-04-09T23:58:28.429 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:28.430 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:28Z" } 2024-04-09T23:58:28.431 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:28.431 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:28.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:28.431 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:33.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:33.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:33.433 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:33.493 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:33.493 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:33.571 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:33.571 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:33.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:33.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:33.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:33.575 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:33Z" 2024-04-09T23:58:33.576 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:33.577 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:33Z" } 2024-04-09T23:58:33.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:33.577 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:33.578 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:33.578 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:38.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:38.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:38.579 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:38.638 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:38.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:38.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:38.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:38.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:38.699 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:38.700 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:38.701 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:38.701 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:38Z" 2024-04-09T23:58:38.701 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:38.702 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:38Z" } 2024-04-09T23:58:38.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:38.702 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:38.702 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:38.702 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:43.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:43.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:43.704 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:43.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:43.764 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:43.847 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:43.848 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:43.849 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:43.850 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:43.851 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:43.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:43.853 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:43Z" 2024-04-09T23:58:43.853 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:43.854 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:43Z" } 2024-04-09T23:58:43.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:43.854 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:43.854 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:43.854 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:48.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:48.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:48.855 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:48.916 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:48.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:48.996 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:48.996 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:48.996 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:48.996 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:48.996 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:48.997 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:48.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:48.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:49.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:49.001 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:48Z" 2024-04-09T23:58:49.002 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:49.003 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:48Z" } 2024-04-09T23:58:49.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:49.003 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:49.004 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:49.004 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:54.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:54.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:54.005 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:54.125 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:54.126 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:54.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:54.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:54.129 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:54Z" 2024-04-09T23:58:54.130 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:54.131 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:54Z" } 2024-04-09T23:58:54.131 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:54.131 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:54.131 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:54.132 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:58:59.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:59.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:58:59.133 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:58:59.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:58:59.194 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:58:59.272 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:58:59.272 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:58:59.272 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:58:59.273 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:58:59.274 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:59.275 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:58:59.276 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:58:59Z" 2024-04-09T23:58:59.277 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:58:59.278 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:58:59Z" } 2024-04-09T23:58:59.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:58:59.279 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:58:59.279 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:58:59.279 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:59:04.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:04.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:04.280 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:04.340 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:04.340 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:04.410 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:04.411 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:04.412 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:04.414 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:04.415 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:59:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:59:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:04.416 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:04Z" 2024-04-09T23:59:04.416 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:04.417 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:59:04Z" } 2024-04-09T23:59:04.417 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:59:04.417 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:04.417 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:59:04.417 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:59:09.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:09.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:09.419 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:09.482 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:09.482 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:09.540 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:09.541 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:09.542 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:09.543 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:09.544 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:59:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:59:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:09.545 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:09Z" 2024-04-09T23:59:09.545 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:09.546 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:59:09Z" } 2024-04-09T23:59:09.546 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:59:09.546 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:09.546 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:59:09.546 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:59:14.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:14.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:14.548 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:14.607 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:14.607 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:14.686 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:14.687 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:14.688 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:14.689 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:59:14.690 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:59:14.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:14.691 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:14Z" 2024-04-09T23:59:14.691 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:14.692 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:59:14Z" } 2024-04-09T23:59:14.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:59:14.692 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:14.692 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:59:14.692 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:59:19.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:19.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:19.693 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:19.753 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:19.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:19.804 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:19.804 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:19.805 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:19.806 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:19.807 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:19.808 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:59:19.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:19.810 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:19Z" 2024-04-09T23:59:19.810 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:19.811 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:59:19Z" } 2024-04-09T23:59:19.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:59:19.811 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:19.811 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:59:19.811 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:59:24.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:24.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:24.812 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:24.872 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:24.873 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:24.945 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:24.945 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:24.945 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:24.945 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:24.945 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:59:24.946 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023809", 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:59:24.947 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:24.948 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:59:24.949 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:24.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:24.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:24.949 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:24.949 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000018901", 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:24.950 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:24Z" 2024-04-09T23:59:24.951 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:24.952 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023809", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:55:50.552935Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018901", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:50.522415Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:59:24Z" } 2024-04-09T23:59:24.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-09T23:59:24.952 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:24.953 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-09T23:59:24.953 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023949 target=00000000000000000000:00000000000000023809, shard=3 master=00000000000000000000:00000000000000019041 target=00000000000000000000:00000000000000018901 2024-04-09T23:59:29.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:29.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:29.954 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:30.014 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.014 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:30.082 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:30.084 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-09T23:59:30.085 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:30.086 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:30.087 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-09T23:59:30.088 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-09T23:59:30.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.088 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:30Z" 2024-04-09T23:59:30.088 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:30.089 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-09T23:59:30Z" } 2024-04-09T23:59:30.089 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-09T23:59:30.089 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:30.089 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-09T23:59:30.089 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-09T23:59:30.090 DEBUG:boto:path=/xryqqk-75/ 2024-04-09T23:59:30.090 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-09T23:59:30.090 DEBUG:boto:Method: HEAD 2024-04-09T23:59:30.090 DEBUG:boto:Path: /xryqqk-75/ 2024-04-09T23:59:30.090 DEBUG:boto:Data: 2024-04-09T23:59:30.090 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.090 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.090 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.090 DEBUG:boto:Params: {} 2024-04-09T23:59:30.091 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-09T23:59:30.091 DEBUG:boto:Token: None 2024-04-09T23:59:30.091 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-75/ 2024-04-09T23:59:30.091 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:+atGoN4bxR5/DV3VJN9q9T4CDIk= 2024-04-09T23:59:30.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+atGoN4bxR5/DV3VJN9q9T4CDIk=', 'Content-Length': '0'} 2024-04-09T23:59:30.099 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f3264460de1d400-006615d662-2ac5-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.099 DEBUG:boto:path=/xryqqk-75/obj1 2024-04-09T23:59:30.099 DEBUG:boto:auth_path=/xryqqk-75/obj1 2024-04-09T23:59:30.099 DEBUG:boto:Method: HEAD 2024-04-09T23:59:30.100 DEBUG:boto:Path: /xryqqk-75/obj1 2024-04-09T23:59:30.100 DEBUG:boto:Data: 2024-04-09T23:59:30.100 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.100 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.100 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.100 DEBUG:boto:Params: {} 2024-04-09T23:59:30.100 DEBUG:boto:Token: None 2024-04-09T23:59:30.100 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-75/obj1 2024-04-09T23:59:30.100 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:l/da04JrOoGqUs4Y6Q8g5w79IsA= 2024-04-09T23:59:30.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:l/da04JrOoGqUs4Y6Q8g5w79IsA=', 'Content-Length': '0'} 2024-04-09T23:59:30.103 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090c2f94c1df053ff-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.103 DEBUG:boto:path=/xryqqk-75/obj1 2024-04-09T23:59:30.103 DEBUG:boto:auth_path=/xryqqk-75/obj1 2024-04-09T23:59:30.103 DEBUG:boto:Method: GET 2024-04-09T23:59:30.103 DEBUG:boto:Path: /xryqqk-75/obj1 2024-04-09T23:59:30.103 DEBUG:boto:Data: 2024-04-09T23:59:30.104 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.104 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.104 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.104 DEBUG:boto:Params: {} 2024-04-09T23:59:30.104 DEBUG:boto:Token: None 2024-04-09T23:59:30.104 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-75/obj1 2024-04-09T23:59:30.104 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:LvlxBi23xVi7jKgDKL0MpI/UhiQ= 2024-04-09T23:59:30.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LvlxBi23xVi7jKgDKL0MpI/UhiQ=', 'Content-Length': '0'} 2024-04-09T23:59:30.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b593ea90adccc85b-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.106 DEBUG:boto:path=/xryqqk-75/obj2 2024-04-09T23:59:30.106 DEBUG:boto:auth_path=/xryqqk-75/obj2 2024-04-09T23:59:30.106 DEBUG:boto:Method: HEAD 2024-04-09T23:59:30.107 DEBUG:boto:Path: /xryqqk-75/obj2 2024-04-09T23:59:30.107 DEBUG:boto:Data: 2024-04-09T23:59:30.107 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.107 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.107 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.107 DEBUG:boto:Params: {} 2024-04-09T23:59:30.107 DEBUG:boto:Token: None 2024-04-09T23:59:30.107 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-75/obj2 2024-04-09T23:59:30.107 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y8FcWJOSuDQBaO19tClIjEmUVQQ= 2024-04-09T23:59:30.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y8FcWJOSuDQBaO19tClIjEmUVQQ=', 'Content-Length': '0'} 2024-04-09T23:59:30.109 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038f60cf2dfc0d4f7-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.109 DEBUG:boto:path=/xryqqk-75/obj2 2024-04-09T23:59:30.109 DEBUG:boto:auth_path=/xryqqk-75/obj2 2024-04-09T23:59:30.110 DEBUG:boto:Method: GET 2024-04-09T23:59:30.110 DEBUG:boto:Path: /xryqqk-75/obj2 2024-04-09T23:59:30.110 DEBUG:boto:Data: 2024-04-09T23:59:30.110 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.111 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.111 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.111 DEBUG:boto:Params: {} 2024-04-09T23:59:30.111 DEBUG:boto:Token: None 2024-04-09T23:59:30.111 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-75/obj2 2024-04-09T23:59:30.111 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:tyD2YVpMI0ys/9IZCFMUSd+nNpE= 2024-04-09T23:59:30.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tyD2YVpMI0ys/9IZCFMUSd+nNpE=', 'Content-Length': '0'} 2024-04-09T23:59:30.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046aa55b51719b466-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.113 DEBUG:boto:path=/xryqqk-76/ 2024-04-09T23:59:30.113 DEBUG:boto:auth_path=/xryqqk-76/ 2024-04-09T23:59:30.114 DEBUG:boto:Method: HEAD 2024-04-09T23:59:30.114 DEBUG:boto:Path: /xryqqk-76/ 2024-04-09T23:59:30.114 DEBUG:boto:Data: 2024-04-09T23:59:30.114 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.114 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.114 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.114 DEBUG:boto:Params: {} 2024-04-09T23:59:30.114 DEBUG:boto:Token: None 2024-04-09T23:59:30.115 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-76/ 2024-04-09T23:59:30.115 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:7QyP5kOkWtrZ7SWYw+HkEKiBf+c= 2024-04-09T23:59:30.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7QyP5kOkWtrZ7SWYw+HkEKiBf+c=', 'Content-Length': '0'} 2024-04-09T23:59:30.119 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b922379a865aed3f-006615d662-2ac5-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.119 DEBUG:boto:path=/xryqqk-76/obj1 2024-04-09T23:59:30.120 DEBUG:boto:auth_path=/xryqqk-76/obj1 2024-04-09T23:59:30.120 DEBUG:boto:Method: HEAD 2024-04-09T23:59:30.120 DEBUG:boto:Path: /xryqqk-76/obj1 2024-04-09T23:59:30.120 DEBUG:boto:Data: 2024-04-09T23:59:30.120 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.120 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.120 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.120 DEBUG:boto:Params: {} 2024-04-09T23:59:30.120 DEBUG:boto:Token: None 2024-04-09T23:59:30.120 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-76/obj1 2024-04-09T23:59:30.121 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:cT5RNBTZpHvyVaaenX3zh2J+f1o= 2024-04-09T23:59:30.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cT5RNBTZpHvyVaaenX3zh2J+f1o=', 'Content-Length': '0'} 2024-04-09T23:59:30.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c58df976c1cf015-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.125 DEBUG:boto:path=/xryqqk-76/obj1 2024-04-09T23:59:30.125 DEBUG:boto:auth_path=/xryqqk-76/obj1 2024-04-09T23:59:30.125 DEBUG:boto:Method: GET 2024-04-09T23:59:30.125 DEBUG:boto:Path: /xryqqk-76/obj1 2024-04-09T23:59:30.125 DEBUG:boto:Data: 2024-04-09T23:59:30.125 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.125 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.126 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.126 DEBUG:boto:Params: {} 2024-04-09T23:59:30.126 DEBUG:boto:Token: None 2024-04-09T23:59:30.126 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-76/obj1 2024-04-09T23:59:30.126 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:bzylftTnS63ddQvGidwU0dKksZA= 2024-04-09T23:59:30.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bzylftTnS63ddQvGidwU0dKksZA=', 'Content-Length': '0'} 2024-04-09T23:59:30.128 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2a3e44861348f6b-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.173 DEBUG:boto:path=/xryqqk-76/obj2 2024-04-09T23:59:30.174 DEBUG:boto:auth_path=/xryqqk-76/obj2 2024-04-09T23:59:30.174 DEBUG:boto:Method: HEAD 2024-04-09T23:59:30.174 DEBUG:boto:Path: /xryqqk-76/obj2 2024-04-09T23:59:30.174 DEBUG:boto:Data: 2024-04-09T23:59:30.174 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.174 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.174 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.174 DEBUG:boto:Params: {} 2024-04-09T23:59:30.175 DEBUG:boto:Token: None 2024-04-09T23:59:30.175 DEBUG:boto:StringToSign: HEAD Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-76/obj2 2024-04-09T23:59:30.175 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:0hOvs2ZbGdodZET5iNe0Z6OW2/U= 2024-04-09T23:59:30.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0hOvs2ZbGdodZET5iNe0Z6OW2/U=', 'Content-Length': '0'} 2024-04-09T23:59:30.178 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d900c5b81f72764-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.178 DEBUG:boto:path=/xryqqk-76/obj2 2024-04-09T23:59:30.178 DEBUG:boto:auth_path=/xryqqk-76/obj2 2024-04-09T23:59:30.178 DEBUG:boto:Method: GET 2024-04-09T23:59:30.178 DEBUG:boto:Path: /xryqqk-76/obj2 2024-04-09T23:59:30.178 DEBUG:boto:Data: 2024-04-09T23:59:30.179 DEBUG:boto:Headers: {} 2024-04-09T23:59:30.179 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-09T23:59:30.179 DEBUG:boto:Port: 8001 2024-04-09T23:59:30.179 DEBUG:boto:Params: {} 2024-04-09T23:59:30.179 DEBUG:boto:Token: None 2024-04-09T23:59:30.179 DEBUG:boto:StringToSign: GET Tue, 09 Apr 2024 23:59:30 GMT /xryqqk-76/obj2 2024-04-09T23:59:30.179 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:5S8o8DWow9e6+VPolSCJa+z1U5Y= 2024-04-09T23:59:30.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5S8o8DWow9e6+VPolSCJa+z1U5Y=', 'Content-Length': '0'} 2024-04-09T23:59:30.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:57:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000509e948ef3dd77a0-006615d662-2ac5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-09T23:59:30.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.186 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:30.243 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.243 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:30.300 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:59:30.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.301 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:30.359 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.359 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:30.401 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: cannot find source zone id for name=* 2024-04-09T23:59:30.401 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: cannot find source zone id for name=* 2024-04-09T23:59:30.419 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-09T23:59:30.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.419 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:30.473 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.473 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:30.540 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:59:30.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.541 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:30.592 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.592 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:30.676 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:30.676 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:30.676 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 18, 2024-04-09T23:59:30.676 INFO:teuthology.orchestra.run.smithi063.stdout: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-09T23:59:30.676 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": [], 2024-04-09T23:59:30.676 INFO:teuthology.orchestra.run.smithi063.stdout: "period_map": { 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "zonegroups": [ 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a", 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": true, 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:59:30.677 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a1", 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.678 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a2", 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:59:30.680 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": false, 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.681 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [ 2024-04-09T23:59:30.682 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-group", 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "data_flow": { 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "symmetrical": [ 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-flow1", 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.683 INFO:teuthology.orchestra.run.smithi063.stdout: "directional": [ 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "source_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "dest_zone": "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "pipes": [ 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-73", 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-73", 2024-04-09T23:59:30.684 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.685 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.686 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-74", 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-74", 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:59:30.687 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:59:30.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-75", 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:59:30.689 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:59:30.690 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe-xryqqk-76", 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-75", 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-09T23:59:30.691 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "xryqqk-76", 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:59:30.692 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "sync-pipe", 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:59:30.693 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "*" 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": { 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket": "*", 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "*" 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "params": { 2024-04-09T23:59:30.694 INFO:teuthology.orchestra.run.smithi063.stdout: "source": { 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: "filter": { 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [] 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: "dest": {}, 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": "system", 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: "user": "" 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.695 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "allowed" 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "b", 2024-04-09T23:59:30.696 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "api_name": "", 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "is_master": false, 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames": [], 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "hostnames_s3website": [], 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "zones": [ 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.697 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b1", 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:59:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b2", 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "endpoints": [ 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-09T23:59:30.699 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.700 INFO:teuthology.orchestra.run.smithi063.stdout: "log_meta": false, 2024-04-09T23:59:30.706 INFO:teuthology.orchestra.run.smithi063.stdout: "log_data": true, 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_index_max_shards": 11, 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "read_only": false, 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tier_type": "", 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from_all": true, 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_from": [], 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_zone": "", 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "supported_features": [ 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "compress-encrypted", 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.707 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "placement_targets": [ 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default-placement", 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_classes": [ 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "STANDARD" 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.708 INFO:teuthology.orchestra.run.smithi063.stdout: "default_placement": "default-placement", 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_policy": { 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "groups": [] 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled_features": [ 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "resharding" 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "short_zone_ids": [ 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.709 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2914326150 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 89109471 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 3383989928 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.710 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 2084093761 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zonegroup": "a", 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "period_config": { 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_quota": { 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:59:30.711 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "user_quota": { 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "check_on_raw": false, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size": -1, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max_size_kb": 0, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max_objects": -1 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "user_ratelimit": { 2024-04-09T23:59:30.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "bucket_ratelimit": { 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:59:30.713 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "anonymous_ratelimit": { 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_ops": 0, 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_ops": 0, 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "max_read_bytes": 0, 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "max_write_bytes": 0, 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": false 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_name": "test-realm", 2024-04-09T23:59:30.714 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:30.715 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:30.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.716 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:30.762 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.762 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:30.829 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:59:30.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:30.830 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket xryqqk-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:30.891 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:31.029 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-09T23:59:31.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.030 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket xryqqk-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:31.109 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:31.109 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:31.268 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-09T23:59:31.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xryqqk-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=xryqqk-75', '--source-bucket=*', '--dest-bucket=xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xryqqk-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=xryqqk-75', '--source-bucket=*', '--dest-bucket=xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-xryqqk-75 --source-zones=a1 --dest-zones=a2 --bucket=xryqqk-75 '--source-bucket=*' --dest-bucket=xryqqk-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:31.340 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:31.340 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:31.463 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-xryqqk-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-09T23:59:31.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xryqqk-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=xryqqk-75', '--source-bucket=*', '--dest-bucket=xryqqk-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-xryqqk-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=xryqqk-75', '--source-bucket=*', '--dest-bucket=xryqqk-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.464 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-xryqqk-76 --source-zones=a1 --dest-zones=a2 --bucket=xryqqk-75 '--source-bucket=*' --dest-bucket=xryqqk-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:31.525 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:31.526 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:31.662 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-xryqqk-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-09T23:59:31.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.663 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket xryqqk-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:31.722 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:31.723 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:31.870 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe-xryqqk-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-09T23:59:31.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-09T23:59:31.871 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-09T23:59:31.924 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:31.925 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:32.023 INFO:teuthology.orchestra.run.smithi063.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-75","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-76","source":{"bucket":"xryqqk-75","zones":["a1"]},"dest":{"bucket":"xryqqk-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-09T23:59:32.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:32.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:32.024 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:32.072 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:32.072 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:32.110 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-09T23:59:32.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707171.812840_1528.1", 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:59:31.812840Z" 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:32.133 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-09T23:59:32.134 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:32.134 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-09T23:59:32.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:32.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:32.135 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:32.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:32.184 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:32.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:32.233+0000 7fc007fb2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:32.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:32.234+0000 7fc007fb2780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:32.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:59:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:59:32.256 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:32.257 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:32.258 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:59:32.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:32.260 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:32Z" 2024-04-09T23:59:32.260 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:32.260 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:59:32Z" } 2024-04-09T23:59:32.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:59:32.261 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:32.261 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:32.261 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707171.812840_1528.1 target=1_1712707044.369051_1496.1 2024-04-09T23:59:37.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:37.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:37.263 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:37.320 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:37.320 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:37.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:37.362+0000 7f26e898e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:37.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:37.363+0000 7f26e898e780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:37.382 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:37.382 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:37.382 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:37.383 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:37.383 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:37.383 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:37.383 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:37.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:37.384 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:37.384 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:37.384 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:37.385 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:37.385 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:37.385 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:37.385 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:37.385 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:59:37.386 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:37.386 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:59:37.386 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:37.386 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:37.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:37.387 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:37.387 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:37.387 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:37.388 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:37.388 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:37.388 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:37.388 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:37.388 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:37.389 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:59:37.389 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:37.389 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:37.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:37.389 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:37.389 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:37.390 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:37.390 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:37.390 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:59:37.390 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:37.390 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:37.391 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:37.391 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:59:37.391 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:37.391 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:37.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:37.392 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:37.392 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:37.392 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:37.392 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:37.392 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:37.393 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:37.393 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:37.393 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:37.393 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:59:37.393 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:37.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:37.394 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:37.394 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:37.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:37.394 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:37.395 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:59:37.395 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:59:37.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:37.395 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:37Z" 2024-04-09T23:59:37.395 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:37.397 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:59:37Z" } 2024-04-09T23:59:37.397 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:59:37.397 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:37.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:37.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707171.812840_1528.1 target=1_1712707044.369051_1496.1 2024-04-09T23:59:42.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:42.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:42.399 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:42.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:42.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:42.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:42.510+0000 7f93a5b57780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:42.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:42.511+0000 7f93a5b57780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:42.533 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:42.534 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:42.535 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:42.536 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:42.537 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:42.538 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:42Z" 2024-04-09T23:59:42.539 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:42.540 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:59:42Z" } 2024-04-09T23:59:42.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:59:42.540 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:42.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:42.541 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707171.812840_1528.1 target=1_1712707044.369051_1496.1 2024-04-09T23:59:47.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:47.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:47.543 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:47.602 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:47.602 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:47.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:47.650+0000 7f6520bb8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:47.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:47.651+0000 7f6520bb8780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:47.668 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:47.668 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:59:47.669 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:47.670 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:59:47.671 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:47.672 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:59:47.673 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:59:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:47.674 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:47Z" 2024-04-09T23:59:47.674 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:47.675 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:59:47Z" } 2024-04-09T23:59:47.675 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:59:47.675 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:47.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:47.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707171.812840_1528.1 target=1_1712707044.369051_1496.1 2024-04-09T23:59:52.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:52.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:52.677 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:52.735 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:52.735 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:52.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:52.778+0000 7faff7c8c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:52.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:52.779+0000 7faff7c8c780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:52.793 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:52.793 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:52.793 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:52.793 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:52.795 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:52.796 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:52.797 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:52.798 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:52.799 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:52.800 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:59:52.800 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:59:52.800 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:52.800 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:52Z" 2024-04-09T23:59:52.800 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:52.801 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:59:52Z" } 2024-04-09T23:59:52.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:59:52.801 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:52.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:52.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707171.812840_1528.1 target=1_1712707044.369051_1496.1 2024-04-09T23:59:57.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:57.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-09T23:59:57.803 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-09T23:59:57.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-09T23:59:57.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-09T23:59:57.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:57.919+0000 7fd2b9dea780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:57.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-09T23:59:57.920+0000 7fd2b9dea780 20 RGW-SYNC:meta: read sync status 2024-04-09T23:59:57.939 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-09T23:59:57.939 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-09T23:59:57.939 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-09T23:59:57.939 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-09T23:59:57.939 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:57.940 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:57.941 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707044.369051_1496.1", 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:57.942 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:24.369051Z", 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-09T23:59:57.943 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-09T23:59:57.944 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-09T23:59:57Z" 2024-04-09T23:59:57.945 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-09T23:59:57.945 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707044.369051_1496.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:24.369051Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-09T23:59:57Z" } 2024-04-09T23:59:57.946 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707044.369051_1496.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:24.369051Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-09T23:59:57.946 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:57.946 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707044.369051_1496.1', 3: '1_1712699220.323078_81.1'} 2024-04-09T23:59:57.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712707171.812840_1528.1 target=1_1712707044.369051_1496.1 2024-04-10T00:00:02.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:02.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:02.948 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:03.006 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:03.007 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:03.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-10T00:00:03.049+0000 7fc14b061780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:00:03.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-10T00:00:03.050+0000 7fc14b061780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:00:03.065 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:03.066 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.067 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:03.067 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.067 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.067 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.068 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:00:03.069 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707171.812840_1528.1", 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.070 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:59:31.812840Z", 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:00:03.071 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.072 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:03.073 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-10T00:00:03.073 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-10T00:00:03.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.073 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:03Z" 2024-04-10T00:00:03.073 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:03.074 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707171.812840_1528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:59:31.812840Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-10T00:00:03Z" } 2024-04-10T00:00:03.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707171.812840_1528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:59:31.812840Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:00:03.074 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:00:03.075 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:00:03.075 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-10T00:00:03.075 DEBUG:boto:path=/xryqqk-75/ 2024-04-10T00:00:03.075 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-10T00:00:03.075 DEBUG:boto:Method: HEAD 2024-04-10T00:00:03.075 DEBUG:boto:Path: /xryqqk-75/ 2024-04-10T00:00:03.076 DEBUG:boto:Data: 2024-04-10T00:00:03.076 DEBUG:boto:Headers: {} 2024-04-10T00:00:03.076 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:00:03.076 DEBUG:boto:Port: 8000 2024-04-10T00:00:03.076 DEBUG:boto:Params: {} 2024-04-10T00:00:03.076 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:00:03.076 DEBUG:boto:Token: None 2024-04-10T00:00:03.077 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:00:03 GMT /xryqqk-75/ 2024-04-10T00:00:03.077 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qgbH04JEgayr5MwjNEPuPGyiByE= 2024-04-10T00:00:03.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:00:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qgbH04JEgayr5MwjNEPuPGyiByE=', 'Content-Length': '0'} 2024-04-10T00:00:03.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087e90006845fac54-006615d683-2ba6-a1'), ('Content-Length', '0'), ('Date', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:00:03.085 DEBUG:boto:path=/xryqqk-75/obj3 2024-04-10T00:00:03.085 DEBUG:boto:auth_path=/xryqqk-75/obj3 2024-04-10T00:00:03.085 DEBUG:boto:Method: PUT 2024-04-10T00:00:03.085 DEBUG:boto:Path: /xryqqk-75/obj3 2024-04-10T00:00:03.085 DEBUG:boto:Data: 2024-04-10T00:00:03.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:00:03.085 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:00:03.085 DEBUG:boto:Port: 8000 2024-04-10T00:00:03.086 DEBUG:boto:Params: {} 2024-04-10T00:00:03.086 DEBUG:boto:Token: None 2024-04-10T00:00:03.086 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 10 Apr 2024 00:00:03 GMT /xryqqk-75/obj3 2024-04-10T00:00:03.086 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:Y3rgvKtt9J9YRwojonQ625NMiEU= 2024-04-10T00:00:03.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 10 Apr 2024 00:00:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y3rgvKtt9J9YRwojonQ625NMiEU='} 2024-04-10T00:00:03.096 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083fe409cc29115ab-006615d683-2ba6-a1'), ('Date', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:00:03.097 DEBUG:boto:path=/xryqqk-75/ 2024-04-10T00:00:03.097 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-10T00:00:03.097 DEBUG:boto:Method: HEAD 2024-04-10T00:00:03.097 DEBUG:boto:Path: /xryqqk-75/ 2024-04-10T00:00:03.097 DEBUG:boto:Data: 2024-04-10T00:00:03.097 DEBUG:boto:Headers: {} 2024-04-10T00:00:03.097 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:00:03.097 DEBUG:boto:Port: 8000 2024-04-10T00:00:03.097 DEBUG:boto:Params: {} 2024-04-10T00:00:03.098 DEBUG:boto:Token: None 2024-04-10T00:00:03.098 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:00:03 GMT /xryqqk-75/ 2024-04-10T00:00:03.098 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:qgbH04JEgayr5MwjNEPuPGyiByE= 2024-04-10T00:00:03.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:00:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qgbH04JEgayr5MwjNEPuPGyiByE=', 'Content-Length': '0'} 2024-04-10T00:00:03.100 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d219a471dab9be3-006615d683-2ba6-a1'), ('Content-Length', '0'), ('Date', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:00:03.100 DEBUG:boto:path=/xryqqk-75/obj4 2024-04-10T00:00:03.100 DEBUG:boto:auth_path=/xryqqk-75/obj4 2024-04-10T00:00:03.100 DEBUG:boto:Method: PUT 2024-04-10T00:00:03.100 DEBUG:boto:Path: /xryqqk-75/obj4 2024-04-10T00:00:03.101 DEBUG:boto:Data: 2024-04-10T00:00:03.101 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:00:03.101 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:00:03.101 DEBUG:boto:Port: 8000 2024-04-10T00:00:03.101 DEBUG:boto:Params: {} 2024-04-10T00:00:03.101 DEBUG:boto:Token: None 2024-04-10T00:00:03.101 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Wed, 10 Apr 2024 00:00:03 GMT /xryqqk-75/obj4 2024-04-10T00:00:03.101 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:VDy77ZNVQcaSdbsI2v/h3VBsUoA= 2024-04-10T00:00:03.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Wed, 10 Apr 2024 00:00:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VDy77ZNVQcaSdbsI2v/h3VBsUoA='} 2024-04-10T00:00:03.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac7eabfd2ea2ec41-006615d683-2ba6-a1'), ('Date', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:00:03.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.110 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:03.159 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:03.160 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:03.214 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:00:03.231 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.232 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707171.812840_1528.1", 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:59:31.812840Z" 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.233 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-10T00:00:03.234 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:00:03.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-10T00:00:03.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.235 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:03.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:03.284 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:03.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-10T00:00:03.332+0000 7fa773e89780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:00:03.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-10T00:00:03.333+0000 7fa773e89780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:00:03.349 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 1, 2024-04-10T00:00:03.350 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:00:03.351 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712707171.812840_1528.1", 2024-04-10T00:00:03.352 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:59:31.812840Z", 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.353 INFO:teuthology.orchestra.run.smithi063.stdout: "state": 1, 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: "realm_epoch": 2 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:03.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.355 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:03.355 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 1, 2024-04-10T00:00:03.355 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 1 2024-04-10T00:00:03.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.355 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:03Z" 2024-04-10T00:00:03.355 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:03.356 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "feef76df-6795-4193-8fee-aa02d98d3595", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712704766.055964_540.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-09T23:19:26.055964Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712704766.074942_528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:19:26.074942Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712707171.812840_1528.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:59:31.812840Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712699220.323078_81.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T21:47:00.323078Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-10T00:00:03Z" } 2024-04-10T00:00:03.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712707171.812840_1528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:59:31.812840Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:00:03.356 DEBUG:rgw_multi.tests:log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:00:03.356 DEBUG:rgw_multi.tests:sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712707171.812840_1528.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:00:03.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-10T00:00:03.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.357 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:03.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:03.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:03.475 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-10T00:00:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.475 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012243", 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-10T00:00:03.110282Z" 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000024090", 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-10T00:00:03.097238Z" 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.476 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: "last_update": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:03.477 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.477 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-10T00:00:03.478 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:03.478 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:00:03.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:03.478 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:03.529 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:03.529 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:03.596 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.597 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.598 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:03.599 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:03.599 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.599 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.600 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:03.601 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:03Z" 2024-04-10T00:00:03.602 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:03.603 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:03Z" } 2024-04-10T00:00:03.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:03.604 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:03.604 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:03.604 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:08.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:08.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:08.605 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:08.728 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:08.728 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:08.729 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:08.730 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:08.731 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:08.732 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:08.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:08.734 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:08Z" 2024-04-10T00:00:08.734 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:08.735 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:08Z" } 2024-04-10T00:00:08.735 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:08.735 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:08.735 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:08.735 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:13.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:13.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:13.737 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:13.799 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:13.799 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:13.879 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:13.879 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:13.879 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:13.880 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:13.881 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:13.882 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:13.883 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:13.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:13.885 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:13Z" 2024-04-10T00:00:13.885 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:13.886 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:13Z" } 2024-04-10T00:00:13.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:13.886 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:13.886 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:13.886 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:18.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:18.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:18.888 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:18.948 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:18.948 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:19.007 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:19.007 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:19.008 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:19.009 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:19.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:19.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:19.012 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:19.013 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:18Z" 2024-04-10T00:00:19.014 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:19.015 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:18Z" } 2024-04-10T00:00:19.015 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:19.015 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:19.016 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:19.016 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:24.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:24.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:24.018 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:24.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:24.078 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:24.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:24.152 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:24.153 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:24.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:24.155 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:24.156 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:24.156 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:24.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:24.156 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:24.157 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:24.158 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:24.158 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:24.159 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:24Z" 2024-04-10T00:00:24.160 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:24.161 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:24Z" } 2024-04-10T00:00:24.161 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:24.161 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:24.161 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:24.161 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:29.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:29.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:29.163 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:29.223 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:29.224 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:29.303 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:29.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:29.305 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:29.307 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:29Z" 2024-04-10T00:00:29.308 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:29.309 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:29Z" } 2024-04-10T00:00:29.309 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:29.309 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:29.309 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:29.310 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:34.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:34.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:34.311 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:34.367 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:34.444 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:34.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:34.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:34.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:34.448 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:34Z" 2024-04-10T00:00:34.449 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:34.450 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:34Z" } 2024-04-10T00:00:34.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:34.450 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:34.451 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:34.451 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:39.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:39.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:39.452 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:39.512 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:39.513 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:39.570 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:39.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:39.572 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:39.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:39.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:39Z" 2024-04-10T00:00:39.575 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:39.576 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:39Z" } 2024-04-10T00:00:39.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:39.577 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:39.577 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:39.577 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:44.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:44.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:44.578 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:44.638 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:44.638 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:44.718 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:44.718 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:44.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:44.721 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:44.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:44.723 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:44Z" 2024-04-10T00:00:44.723 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:44.724 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:44Z" } 2024-04-10T00:00:44.724 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:44.724 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:44.724 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:44.724 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:49.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:49.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:49.726 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:49.785 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:49.785 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:49.853 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:49.854 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:49.855 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:49.856 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:49.857 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:49.858 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:49.859 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:49.859 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:49.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:49.859 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:49Z" 2024-04-10T00:00:49.859 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:49.860 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:49Z" } 2024-04-10T00:00:49.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:49.860 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:49.860 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:49.860 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:54.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:54.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:54.862 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:00:54.925 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:00:54.925 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:00:54.983 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:54.984 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:00:54.985 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:00:54.986 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:00:54.987 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:00:54.988 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:00:54.989 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:00:54.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:00:54.989 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:00:54Z" 2024-04-10T00:00:54.989 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:00:54.990 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:00:54Z" } 2024-04-10T00:00:54.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:00:54.990 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:54.990 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:00:54.990 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:00:59.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:59.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:00:59.991 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:00.051 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:00.051 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:00.131 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:00.131 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:00.131 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:00.132 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:00.133 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:00.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:00.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:00.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:00.134 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:00.134 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:00.134 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:00.135 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:00.136 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:00.136 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:00.136 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:00.136 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:00.136 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:00.136 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:00.137 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:00.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:00.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:00.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:00.138 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:00.138 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:00.138 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:00.139 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:00.139 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:00.139 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:00.139 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:00.139 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:00.139 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:00.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:00.140 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:00.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:00.140 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:00.140 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:00.140 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:00.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:00.141 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:00Z" 2024-04-10T00:01:00.141 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:00.141 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:00Z" } 2024-04-10T00:01:00.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:00.142 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:00.142 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:00.142 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:05.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:05.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:05.143 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:05.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:05.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:05.277 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:05.278 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:05.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:05.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:05.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:05Z" 2024-04-10T00:01:05.282 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:05.283 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:05Z" } 2024-04-10T00:01:05.283 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:05.284 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:05.284 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:05.284 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:10.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:10.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:10.286 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:10.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:10.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:10.403 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:10.403 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:10.403 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:10.404 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:10.405 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:10.406 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:10.407 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:10.408 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:10.409 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:10.409 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:10Z" 2024-04-10T00:01:10.409 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:10.410 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:10Z" } 2024-04-10T00:01:10.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:10.410 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:10.410 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:10.411 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:15.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:15.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:15.412 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:15.555 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:15Z" 2024-04-10T00:01:15.561 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:15.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:15Z" } 2024-04-10T00:01:15.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:15.561 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:15.561 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:15.561 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:20.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:20.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:20.563 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:20.624 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:20.625 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:20.681 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:20Z" 2024-04-10T00:01:20.689 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:20.690 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:20Z" } 2024-04-10T00:01:20.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:20.690 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:20.690 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:20.690 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:25.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:25.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:25.692 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:25.752 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:25.752 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:25.813 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:25.813 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:25.813 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:25.814 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:25.815 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:25.816 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:25.817 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:25.817 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:25.817 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:25.817 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:25.817 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:25.817 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:25.818 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:25.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:25.820 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:25Z" 2024-04-10T00:01:25.820 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:25.823 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:25Z" } 2024-04-10T00:01:25.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:25.823 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:25.824 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:25.824 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:30.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:30.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:30.826 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:30.886 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:30.887 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:30.954 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:30.954 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:30.954 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:30.955 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:30.955 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:30.955 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:30.955 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:30.955 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:30.957 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:30.957 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:30.957 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:30.958 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000023949", 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.805118Z" 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:30.959 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:30.960 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:30.961 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:30Z" 2024-04-10T00:01:30.962 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:30.963 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012103", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-09T23:55:50.606072Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023949", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-09T23:57:56.805118Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:30Z" } 2024-04-10T00:01:30.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023949', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.805118Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:30.963 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:30.963 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023949', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:30.963 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012243 target=00000000000000000000:00000000000000012103, shard=1 master=00000000000000000000:00000000000000024090 target=00000000000000000000:00000000000000023949 2024-04-10T00:01:35.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:35.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:35.965 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:36.025 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.025 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.088 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-10T00:01:36.088 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_status": { 2024-04-10T00:01:36.089 INFO:teuthology.orchestra.run.smithi063.stdout: "info": { 2024-04-10T00:01:36.089 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "sync", 2024-04-10T00:01:36.089 INFO:teuthology.orchestra.run.smithi063.stdout: "num_shards": 4, 2024-04-10T00:01:36.089 INFO:teuthology.orchestra.run.smithi063.stdout: "instance_id": 11300205680425452250 2024-04-10T00:01:36.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:36.089 INFO:teuthology.orchestra.run.smithi063.stdout: "markers": [ 2024-04-10T00:01:36.090 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:36.090 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 0, 2024-04-10T00:01:36.090 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:36.090 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:36.090 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000012243", 2024-04-10T00:01:36.090 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:36.091 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 9, 2024-04-10T00:01:36.091 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:36.091 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-10T00:00:03.110282Z" 2024-04-10T00:01:36.091 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:36.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:36.091 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:36.092 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 1, 2024-04-10T00:01:36.092 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:36.092 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:36.093 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000024090", 2024-04-10T00:01:36.093 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:36.093 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 57, 2024-04-10T00:01:36.093 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:36.093 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-10T00:00:03.097238Z" 2024-04-10T00:01:36.094 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:36.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:36.094 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:36.094 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 2, 2024-04-10T00:01:36.095 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:36.095 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:36.095 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:36.095 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:36.095 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:36.095 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "key": 3, 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "val": { 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "status": "incremental-sync", 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "marker": "00000000000000000000:00000000000000019041", 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "next_step_marker": "", 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "total_entries": 0, 2024-04-10T00:01:36.096 INFO:teuthology.orchestra.run.smithi063.stdout: "pos": 0, 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: "timestamp": "2024-04-09T23:57:56.834128Z" 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: "full_sync": { 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: "total": 66, 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: "complete": 66 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout: "current_time": "2024-04-10T00:01:36Z" 2024-04-10T00:01:36.097 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-10T00:01:36.099 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 11300205680425452250 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012243", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-10T00:00:03.110282Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000024090", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-10T00:00:03.097238Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019863", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:55:21.139072Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019041", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-09T23:57:56.834128Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-10T00:01:36Z" } 2024-04-10T00:01:36.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012243', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-10T00:00:03.110282Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000024090', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-10T00:00:03.097238Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019041', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:57:56.834128Z'}}] 2024-04-10T00:01:36.099 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:36.099 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012243', 1: '00000000000000000000:00000000000000024090', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000019041'} 2024-04-10T00:01:36.099 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:36.100 DEBUG:boto:path=/xryqqk-75/ 2024-04-10T00:01:36.100 DEBUG:boto:auth_path=/xryqqk-75/ 2024-04-10T00:01:36.100 DEBUG:boto:Method: HEAD 2024-04-10T00:01:36.100 DEBUG:boto:Path: /xryqqk-75/ 2024-04-10T00:01:36.100 DEBUG:boto:Data: 2024-04-10T00:01:36.100 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.101 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.101 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.101 DEBUG:boto:Params: {} 2024-04-10T00:01:36.101 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:36.101 DEBUG:boto:Token: None 2024-04-10T00:01:36.102 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-75/ 2024-04-10T00:01:36.102 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:9w1+um3deC76VCOXQdYfL7RMmo0= 2024-04-10T00:01:36.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9w1+um3deC76VCOXQdYfL7RMmo0=', 'Content-Length': '0'} 2024-04-10T00:01:36.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff92c27779f33554-006615d6e0-2ba8-a2'), ('Content-Length', '0'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.108 DEBUG:boto:path=/xryqqk-75/obj3 2024-04-10T00:01:36.108 DEBUG:boto:auth_path=/xryqqk-75/obj3 2024-04-10T00:01:36.108 DEBUG:boto:Method: HEAD 2024-04-10T00:01:36.109 DEBUG:boto:Path: /xryqqk-75/obj3 2024-04-10T00:01:36.109 DEBUG:boto:Data: 2024-04-10T00:01:36.109 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.109 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.109 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.109 DEBUG:boto:Params: {} 2024-04-10T00:01:36.109 DEBUG:boto:Token: None 2024-04-10T00:01:36.109 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-75/obj3 2024-04-10T00:01:36.109 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:MX8RfknIzwxy2WC70SE7AkF4Q1c= 2024-04-10T00:01:36.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MX8RfknIzwxy2WC70SE7AkF4Q1c=', 'Content-Length': '0'} 2024-04-10T00:01:36.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001baf906a57a69c55-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.111 DEBUG:boto:path=/xryqqk-75/obj3 2024-04-10T00:01:36.112 DEBUG:boto:auth_path=/xryqqk-75/obj3 2024-04-10T00:01:36.112 DEBUG:boto:Method: GET 2024-04-10T00:01:36.112 DEBUG:boto:Path: /xryqqk-75/obj3 2024-04-10T00:01:36.112 DEBUG:boto:Data: 2024-04-10T00:01:36.112 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.112 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.112 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.112 DEBUG:boto:Params: {} 2024-04-10T00:01:36.113 DEBUG:boto:Token: None 2024-04-10T00:01:36.113 DEBUG:boto:StringToSign: GET Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-75/obj3 2024-04-10T00:01:36.113 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:dhO+Rk7Jwku4tQlYDrC9fAi5Gzw= 2024-04-10T00:01:36.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dhO+Rk7Jwku4tQlYDrC9fAi5Gzw=', 'Content-Length': '0'} 2024-04-10T00:01:36.115 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044d9b6ca4f79819b-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.115 DEBUG:boto:path=/xryqqk-75/obj4 2024-04-10T00:01:36.116 DEBUG:boto:auth_path=/xryqqk-75/obj4 2024-04-10T00:01:36.116 DEBUG:boto:Method: HEAD 2024-04-10T00:01:36.116 DEBUG:boto:Path: /xryqqk-75/obj4 2024-04-10T00:01:36.116 DEBUG:boto:Data: 2024-04-10T00:01:36.116 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.116 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.116 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.116 DEBUG:boto:Params: {} 2024-04-10T00:01:36.116 DEBUG:boto:Token: None 2024-04-10T00:01:36.117 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-75/obj4 2024-04-10T00:01:36.117 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jKnV1/wQW0eTWtaLSjiPkV4cA5A= 2024-04-10T00:01:36.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jKnV1/wQW0eTWtaLSjiPkV4cA5A=', 'Content-Length': '0'} 2024-04-10T00:01:36.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0e990a08889a8bb-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.119 DEBUG:boto:path=/xryqqk-75/obj4 2024-04-10T00:01:36.119 DEBUG:boto:auth_path=/xryqqk-75/obj4 2024-04-10T00:01:36.119 DEBUG:boto:Method: GET 2024-04-10T00:01:36.119 DEBUG:boto:Path: /xryqqk-75/obj4 2024-04-10T00:01:36.119 DEBUG:boto:Data: 2024-04-10T00:01:36.119 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.119 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.119 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.119 DEBUG:boto:Params: {} 2024-04-10T00:01:36.119 DEBUG:boto:Token: None 2024-04-10T00:01:36.120 DEBUG:boto:StringToSign: GET Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-75/obj4 2024-04-10T00:01:36.120 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:oCcUiaxqLJE1+VcuhXic1Y+vXB4= 2024-04-10T00:01:36.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oCcUiaxqLJE1+VcuhXic1Y+vXB4=', 'Content-Length': '0'} 2024-04-10T00:01:36.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a6f098555f81bcb-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.166 DEBUG:boto:path=/xryqqk-76/ 2024-04-10T00:01:36.167 DEBUG:boto:auth_path=/xryqqk-76/ 2024-04-10T00:01:36.167 DEBUG:boto:Method: HEAD 2024-04-10T00:01:36.167 DEBUG:boto:Path: /xryqqk-76/ 2024-04-10T00:01:36.167 DEBUG:boto:Data: 2024-04-10T00:01:36.167 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.167 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.167 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.167 DEBUG:boto:Params: {} 2024-04-10T00:01:36.167 DEBUG:boto:Token: None 2024-04-10T00:01:36.168 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-76/ 2024-04-10T00:01:36.168 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:/wbVV5kdtpiOLeFcbqAavkQgmZw= 2024-04-10T00:01:36.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/wbVV5kdtpiOLeFcbqAavkQgmZw=', 'Content-Length': '0'} 2024-04-10T00:01:36.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000257050d7cbd64e7d-006615d6e0-2ba8-a2'), ('Content-Length', '0'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.170 DEBUG:boto:path=/xryqqk-76/obj3 2024-04-10T00:01:36.170 DEBUG:boto:auth_path=/xryqqk-76/obj3 2024-04-10T00:01:36.170 DEBUG:boto:Method: HEAD 2024-04-10T00:01:36.170 DEBUG:boto:Path: /xryqqk-76/obj3 2024-04-10T00:01:36.170 DEBUG:boto:Data: 2024-04-10T00:01:36.170 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.170 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.171 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.171 DEBUG:boto:Params: {} 2024-04-10T00:01:36.171 DEBUG:boto:Token: None 2024-04-10T00:01:36.171 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-76/obj3 2024-04-10T00:01:36.171 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:1HBW16srgO0sAyVbSsZL6NFtxcY= 2024-04-10T00:01:36.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1HBW16srgO0sAyVbSsZL6NFtxcY=', 'Content-Length': '0'} 2024-04-10T00:01:36.172 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092640aaa8262b67c-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.173 DEBUG:boto:path=/xryqqk-76/obj3 2024-04-10T00:01:36.173 DEBUG:boto:auth_path=/xryqqk-76/obj3 2024-04-10T00:01:36.173 DEBUG:boto:Method: GET 2024-04-10T00:01:36.173 DEBUG:boto:Path: /xryqqk-76/obj3 2024-04-10T00:01:36.173 DEBUG:boto:Data: 2024-04-10T00:01:36.173 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.173 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.173 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.174 DEBUG:boto:Params: {} 2024-04-10T00:01:36.174 DEBUG:boto:Token: None 2024-04-10T00:01:36.174 DEBUG:boto:StringToSign: GET Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-76/obj3 2024-04-10T00:01:36.174 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:kXQO0GPr8gGsz6Mr+UJEqLykeCY= 2024-04-10T00:01:36.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kXQO0GPr8gGsz6Mr+UJEqLykeCY=', 'Content-Length': '0'} 2024-04-10T00:01:36.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008172aee31aa706f0-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.176 DEBUG:boto:path=/xryqqk-76/obj4 2024-04-10T00:01:36.176 DEBUG:boto:auth_path=/xryqqk-76/obj4 2024-04-10T00:01:36.176 DEBUG:boto:Method: HEAD 2024-04-10T00:01:36.176 DEBUG:boto:Path: /xryqqk-76/obj4 2024-04-10T00:01:36.177 DEBUG:boto:Data: 2024-04-10T00:01:36.177 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.177 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.177 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.177 DEBUG:boto:Params: {} 2024-04-10T00:01:36.177 DEBUG:boto:Token: None 2024-04-10T00:01:36.177 DEBUG:boto:StringToSign: HEAD Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-76/obj4 2024-04-10T00:01:36.177 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:jWW8mZZXUHmfto2O+QNwFZlpkzo= 2024-04-10T00:01:36.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jWW8mZZXUHmfto2O+QNwFZlpkzo=', 'Content-Length': '0'} 2024-04-10T00:01:36.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9fd3394b499a892-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.179 DEBUG:boto:path=/xryqqk-76/obj4 2024-04-10T00:01:36.179 DEBUG:boto:auth_path=/xryqqk-76/obj4 2024-04-10T00:01:36.179 DEBUG:boto:Method: GET 2024-04-10T00:01:36.180 DEBUG:boto:Path: /xryqqk-76/obj4 2024-04-10T00:01:36.180 DEBUG:boto:Data: 2024-04-10T00:01:36.180 DEBUG:boto:Headers: {} 2024-04-10T00:01:36.180 DEBUG:boto:Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.180 DEBUG:boto:Port: 8001 2024-04-10T00:01:36.180 DEBUG:boto:Params: {} 2024-04-10T00:01:36.180 DEBUG:boto:Token: None 2024-04-10T00:01:36.180 DEBUG:boto:StringToSign: GET Wed, 10 Apr 2024 00:01:36 GMT /xryqqk-76/obj4 2024-04-10T00:01:36.180 DEBUG:boto:Signature: AWS N5OV6KWZ8WWR3JMWMG4W:OvcGyQ/Rb0WD6g7x5EYSV9fFP7k= 2024-04-10T00:01:36.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Wed, 10 Apr 2024 00:01:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OvcGyQ/Rb0WD6g7x5EYSV9fFP7k=', 'Content-Length': '0'} 2024-04-10T00:01:36.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Wed, 10 Apr 2024 00:00:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-75:a5723776-f142-428c-a6df-d7bf949257da.10610.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001a8bb9be2c708972-006615d6e0-2ba8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Wed, 10 Apr 2024 00:01:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:36.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'xryqqk-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:36.183 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket xryqqk-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:36.233 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.233 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.301 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-10T00:01:36.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:36.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:36.301 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:36.348 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.429 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_policy":{"groups":[]}} 2024-04-10T00:01:36.430 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2024-04-10T00:01:36.431 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.431 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:36.431 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_encrypted_object_sync 2024-04-10T00:01:36.431 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1319, in test_encrypted_object_sync 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests: eq(data, key.get_contents_as_string(headers=sse_c_headers, encoding='ascii')) 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:AttributeError: 'NoneType' object has no attribute 'get_contents_as_string' 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-54 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-54/ 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-54/ 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-54/ 2024-04-10T00:01:36.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:32 GMT 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:/xryqqk-54/ 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.433 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:k+bqStpueuY03qZzzvaMh2XJPPY= 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k+bqStpueuY03qZzzvaMh2XJPPY=', 'Content-Length': '0'} 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000659d350fd199f573-006615c5fc-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:32 GMT 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:/xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OC1L02YjI3oQSuLzEh5+wDilYT0= 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OC1L02YjI3oQSuLzEh5+wDilYT0='} 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx0000004d5d0b6cf9bfedc-006615c5fc-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-54/testobj-sse-kms 2024-04-10T00:01:36.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-54/testobj-sse-kms 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-54/testobj-sse-kms 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.436 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:32 GMT 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption:aws:kms 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-aws-kms-key-id:testkey-1 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:/xryqqk-54/testobj-sse-kms 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UZSJ65V85toqhO6C96SdsnDjhQc= 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UZSJ65V85toqhO6C96SdsnDjhQc='} 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6eHJ5cXFrLTU0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000000d154ce0ae7f6976-006615c5fc-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702972.930686_822.1", 2024-04-10T00:01:36.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:49:32.930686Z" 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702972.930686_822.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:33.165+0000 7f453bf35780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:33.166+0000 7f453bf35780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:36.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:36.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702972.930686_822.1", 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:49:32.930686Z", 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:49:33Z" 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:36.444 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.445 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1712702972.930686_822.1", 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:49:32.930686Z", 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.448 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:49:33Z" 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702972.930686_822.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:49:32.930686Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702972.930686_822.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702972.930686_822.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-54 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-54/ 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-54/ 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-54/ 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:/xryqqk-54/ 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3y7iItu0wmxzkSlgjr0Mq8ZNKbc= 2024-04-10T00:01:36.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3y7iItu0wmxzkSlgjr0Mq8ZNKbc=', 'Content-Length': '0'} 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abcd83ed53ed6891-006615c5fd-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.452 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-algorithm:AES256 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:/xryqqk-54/testobj-sse-c 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+I/xrDNwSO5R1gmQfwsiScJ8FCk= 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+I/xrDNwSO5R1gmQfwsiScJ8FCk=', 'Content-Length': '0'} 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000949d09b5515f12f2-006615c5fd-1b6f-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:36.453 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.454 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:36.454 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-04-10T00:01:36.454 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:36.457 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:36.457 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:36.457 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:36.457 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-04-10T00:01:36.457 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-11 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:10j7qJF13Kx+xwpmJzleXPyHkb4= 2024-04-10T00:01:36.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:10j7qJF13Kx+xwpmJzleXPyHkb4=', 'Content-Length': '0'} 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d20befe8533abaa6-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-12 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.460 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cojRhJDO97b6Nz4avCW2M+6I+08= 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cojRhJDO97b6Nz4avCW2M+6I+08=', 'Content-Length': '0'} 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002885a5d51a7e9b6e-006615b791-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025b429c117727ec2-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.463 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vpTPsdL1OnbXm7JnjkGJ10cjJrk= 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vpTPsdL1OnbXm7JnjkGJ10cjJrk='} 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029326e52b708c8e6-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdf4f4b2c20924c7-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.466 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F6NRhpBTU00qVRGFdXggzwPpawQ= 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F6NRhpBTU00qVRGFdXggzwPpawQ='} 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f7318b97cec7407-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019e9f18bb43ad2b3-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.469 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6xiW1cB3GcbfM4RG/Q6GfnztTF0= 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6xiW1cB3GcbfM4RG/Q6GfnztTF0='} 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065bf6e16dead069c-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.471 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c993d897ec95c331-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.472 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e6bxfbZplwvKbpX42mJCsfpfL6w= 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e6bxfbZplwvKbpX42mJCsfpfL6w='} 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce68e4ef577c1ff1-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000003e7e23149c6bad1-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.475 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:KQNYALpg5l7NLKj1K73O1ziRzrI= 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KQNYALpg5l7NLKj1K73O1ziRzrI='} 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfdee55f85d13fc0-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008aaf64cc574a2269-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.478 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:j+H8oD+bf2pfPcQwiVGhgKVmuuc= 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:j+H8oD+bf2pfPcQwiVGhgKVmuuc='} 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040627534fe71074f-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fac9dd3bcaae67c-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.481 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hir6sQWp0BpbNY538eiI5IMc43E= 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hir6sQWp0BpbNY538eiI5IMc43E='} 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c71974859424078-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3860e573e1a5540-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.484 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Ryve/BpyvJRD6dMxc0+b1uzDbpM= 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ryve/BpyvJRD6dMxc0+b1uzDbpM='} 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033a87e22a740d5c1-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f936749b676d1936-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.487 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:PShGQ9TTpFC0PA1qsHHHNBf8yj4= 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PShGQ9TTpFC0PA1qsHHHNBf8yj4='} 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040bb5d5196eaf70d-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4= 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WxK7kGcNTbIs1drThzu6BQUi+o4=', 'Content-Length': '0'} 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000cff8259818fc95a-006615b791-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:01 GMT 2024-04-10T00:01:36.490 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yQP2Xzi1FWkpJ1AlXgmDZ7Okc4k= 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:01 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yQP2Xzi1FWkpJ1AlXgmDZ7Okc4k='} 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbc754a1c7f13139-006615b791-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005db5743ef0f4d50-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.493 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:fIRPB5AKIYqRWEKNl9FVFsAvfKo= 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fIRPB5AKIYqRWEKNl9FVFsAvfKo='} 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c47a5ac167dea7e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009eef1d8e4d40e542-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.496 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:jmAvJaBKC44G2ay2WKUMGPR15bw= 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jmAvJaBKC44G2ay2WKUMGPR15bw='} 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eb8ff2acc0ce2f1f-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b7d25509c46b144-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13 2024-04-10T00:01:36.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Ffa6RaCpPYGKkNcbTEk37Ok8tsY= 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ffa6RaCpPYGKkNcbTEk37Ok8tsY='} 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce2eddf29522a393-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a0bf89b47aee2d2-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14 2024-04-10T00:01:36.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZJfB8UL7vvr/mMpI1WWTksH32r8= 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZJfB8UL7vvr/mMpI1WWTksH32r8='} 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000408844e562a661fb-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000176d9c305c9c5133-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e5GEuEUjNumj/b0l65qcMA3tg38= 2024-04-10T00:01:36.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e5GEuEUjNumj/b0l65qcMA3tg38='} 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1ee8e4981c8880a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.506 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9355467d39f6fd2-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QiRHeWodpao5H61ACvGukDEm6TU= 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QiRHeWodpao5H61ACvGukDEm6TU='} 2024-04-10T00:01:36.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008a3754e24471d0e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.509 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cff02e3403155ccf-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gyfRG+7NLSmPngX/MFTzJAPTJh4= 2024-04-10T00:01:36.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gyfRG+7NLSmPngX/MFTzJAPTJh4='} 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b9b560c3fef51055-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.512 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002844b45bcc27a18c-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e5d71+O0LTiXoseyzYABUn0QsMs= 2024-04-10T00:01:36.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e5d71+O0LTiXoseyzYABUn0QsMs='} 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005ba579102acca70-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eda08343d3b0709-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:36.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:36.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19 2024-04-10T00:01:36.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:NW2pk/My0osbA/0ryq4uV97Stis= 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NW2pk/My0osbA/0ryq4uV97Stis='} 2024-04-10T00:01:36.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc78162f0e13d8ce-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.519 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000162f797834e21ebd-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YHyAXfMabPoExf7VVXW1XMqJvFo= 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YHyAXfMabPoExf7VVXW1XMqJvFo='} 2024-04-10T00:01:36.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000080ac41ddd781b20d-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.522 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8947376b201bddc-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:boSJVHq3KZZANb35zwwtXHXRoU4= 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:boSJVHq3KZZANb35zwwtXHXRoU4='} 2024-04-10T00:01:36.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f5ff2b8aaf3437f-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.525 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078f4224f64225dca-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22 2024-04-10T00:01:36.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ngCUYHDgGDDwslHo7/zNuoMEIq8= 2024-04-10T00:01:36.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ngCUYHDgGDDwslHo7/zNuoMEIq8='} 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001af5daba99ff485c-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.528 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e24826b8d4af5512-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6PgDHHc+Da/EF7VisCsXUKMZY08= 2024-04-10T00:01:36.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6PgDHHc+Da/EF7VisCsXUKMZY08='} 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ce88978399280c3-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c5fd74d7ee1a620-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:87n9HNK3MVMFp31BYAT9rN/lHRM= 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:87n9HNK3MVMFp31BYAT9rN/lHRM='} 2024-04-10T00:01:36.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7bf423f5c33230e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.534 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c3be54a0fd0781-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4vMVtiUaefthfLa/RvS/o53P1qc= 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4vMVtiUaefthfLa/RvS/o53P1qc='} 2024-04-10T00:01:36.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007732e9ee026e2b4e-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.537 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097c5acbe868bd1c9-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kr6v4spmRTbO0NB4bCRFoPB/rvg= 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kr6v4spmRTbO0NB4bCRFoPB/rvg='} 2024-04-10T00:01:36.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7b0d7f517a49c05-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.540 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043b1e9101bb1daa8-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AUPavD9oQE0diThNvjteOZNERQY= 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AUPavD9oQE0diThNvjteOZNERQY='} 2024-04-10T00:01:36.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000803c044d478ef455-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007329547d70306325-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mviJHRdWk1Nq01BeZB5o9Ol7UGM= 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mviJHRdWk1Nq01BeZB5o9Ol7UGM='} 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094fd2e749ff76f8a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000969af379d364354a-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Thuz2a40OQgl2zPHs7uEjvOYPMY= 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Thuz2a40OQgl2zPHs7uEjvOYPMY='} 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d86754716e3462b-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090b9e7ccf75d4840-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.550 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:dFhywgTfHI1/Qt1mnj/vhMNT87g= 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dFhywgTfHI1/Qt1mnj/vhMNT87g='} 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001829af83d0dda297-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a236f78904915c2e-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.553 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0O9MYwuSQk9BZLQZx2JfrspilSs= 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0O9MYwuSQk9BZLQZx2JfrspilSs='} 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c502e6f96f1cf9a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079cfd1d80e75f5a0-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.556 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:eOnbU1Pe66azANzN4c6FDGdNyzE= 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eOnbU1Pe66azANzN4c6FDGdNyzE='} 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055e8e98684d4b318-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090ea7c50e167c29a-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.559 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:jw3/wO2AO4EACF95Dbeb4KCTIrA= 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jw3/wO2AO4EACF95Dbeb4KCTIrA='} 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d59dbd04f7559e0c-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000430ed0b07713efe6-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:36.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34 2024-04-10T00:01:36.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:avRi6ktHqLAqcMbeHFsTRwjcPWU= 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:avRi6ktHqLAqcMbeHFsTRwjcPWU='} 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cfcdfa9278c6cb7d-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d06d2d5f1dca20bd-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.565 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4ReLLjZgacwyARv4VgAPnyMeiD0= 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4ReLLjZgacwyARv4VgAPnyMeiD0='} 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a412cbd90bd9c82-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008d35acbcc8eac05f-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.568 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AR414Hhi2/5cS4Sb1rr2oUH8DWQ= 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AR414Hhi2/5cS4Sb1rr2oUH8DWQ='} 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a06dd4409382b30-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000408f4efe7eda59e2-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.571 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:fXIrZo6WwTt6RxorL1uCORDQ+2E= 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fXIrZo6WwTt6RxorL1uCORDQ+2E='} 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47897bf408656b2-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.572 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed8b45a16ee78afa-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38 2024-04-10T00:01:36.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+w71DItf/rAeXU4RXFWO1xFZaA8= 2024-04-10T00:01:36.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+w71DItf/rAeXU4RXFWO1xFZaA8='} 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e493b245a6f78d3-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.575 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ece3e40a97950fa8-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39 2024-04-10T00:01:36.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wBKHZL47USiewqblcFyq/hajjCs= 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wBKHZL47USiewqblcFyq/hajjCs='} 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a0493d07cf71a2a-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3e5d89f7e4cf1e9-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40 2024-04-10T00:01:36.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:b/QIDwGtc/jm7yQ5YwgcYcOF/p8= 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b/QIDwGtc/jm7yQ5YwgcYcOF/p8='} 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000057534148d87d2ac-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c4d5c46c49666cc-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41 2024-04-10T00:01:36.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:V6bjX4JBZqef8h9wYPLW+wCvzu8= 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:V6bjX4JBZqef8h9wYPLW+wCvzu8='} 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051e0446cedd8e916-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064de43f5e507f52b-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:36.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.586 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EwV603UV8+D4s4/OBeeLs9g+iwI= 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwV603UV8+D4s4/OBeeLs9g+iwI='} 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f77c6e5cb6754de0-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020a436ff779d077c-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:36.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43 2024-04-10T00:01:36.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9m5oqFeNohkfqkzXJJ9K5oWv4Lo= 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9m5oqFeNohkfqkzXJJ9K5oWv4Lo='} 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069b4351f4547eca0-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009203c8d624dff12c-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:36.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.592 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7ubf6OvWfibIb4HoKG3tqvneB+M= 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ubf6OvWfibIb4HoKG3tqvneB+M='} 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0b0786880a39dd8-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e18a4c1976ee83b4-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:J1nuQkVEP4qhdgFfWSw0+StPQLo= 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J1nuQkVEP4qhdgFfWSw0+StPQLo='} 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ab2c12310821055-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.597 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008ddc4e58bdca595-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cJzKjVa04F3GAP6GHz+SV2G9Y7s= 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cJzKjVa04F3GAP6GHz+SV2G9Y7s='} 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000098d7a9628a141c-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088f5f998f5d5c941-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:RSeMqafVY9cxdOj/dH9f4EDbxfE= 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RSeMqafVY9cxdOj/dH9f4EDbxfE='} 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054319cdfb99f55b5-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.603 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086a1f29cc9f88c60-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.604 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yOq0IY99m820Xsachf2BJa26PLM= 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yOq0IY99m820Xsachf2BJa26PLM='} 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e6737721b6a0e1-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ= 2024-04-10T00:01:36.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ooDsl3kEgyQyar+rTEQW2npSkjQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3cc64815086bbac-006615b792-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.607 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:w40feBMpHqA2Q/11CAwEMx/OakY= 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w40feBMpHqA2Q/11CAwEMx/OakY='} 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000e341eb0c6c917d-006615b792-1110-a1'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q= 2024-04-10T00:01:36.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f4ea8e66d2c808b-006615b792-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj1 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj1 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj1 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.610 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj1 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uzsMPj6IScWCB0t+YwseC7lmK4o= 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uzsMPj6IScWCB0t+YwseC7lmK4o='} 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000876b00436612f475-006615b792-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q= 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f771ea19d1bcc977-006615b792-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj2 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj2 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj2 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj2 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:NWAQhbK+xilZ8EV57XL56PzimbQ= 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:NWAQhbK+xilZ8EV57XL56PzimbQ='} 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ce298dd4db9e7ce-006615b792-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.615 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q= 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UkRd/ktNZk6SbqW9Z+MuF+wHS0Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001991d7cd0597ec6-006615b792-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj3 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj3 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj3 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:02 GMT 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj3 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8YqyamScLXaPibTjVDkCsmY/qq0= 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8YqyamScLXaPibTjVDkCsmY/qq0='} 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004acc6c833a9fba2f-006615b792-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.618 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac7dc78e397c172a-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj4 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj4 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj4 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj4 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SvAjPt495vAKiKkeoicsE8O4iZU= 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SvAjPt495vAKiKkeoicsE8O4iZU='} 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d66bcb5d243ccc4f-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.621 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f3213a4a4877e58-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj5 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj5 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj5 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj5 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Y8pMymnTsYDpmJRDGTpCVuaMnY4= 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y8pMymnTsYDpmJRDGTpCVuaMnY4='} 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba5523ccedf10c93-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.624 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afeee7711b807b50-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj6 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj6 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj6 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj6 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kPxmNw3aa4zbKhaPLZ2as9IDig0= 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kPxmNw3aa4zbKhaPLZ2as9IDig0='} 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e535f02d5b823140-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030feb54700647113-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj7 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj7 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj7 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj7 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tPg7fGt8AesdSSpWKkKdqIcWRjg= 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tPg7fGt8AesdSSpWKkKdqIcWRjg='} 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e614e44132dafbed-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.630 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000568ae60217dcef2-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj8 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj8 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj8 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.631 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj8 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+g6BNPvY+nlblajGcQOdUYzvGGs= 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+g6BNPvY+nlblajGcQOdUYzvGGs='} 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095813fb8dafa7b5f-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eec9b9013ffec45b-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj9 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj9 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj9 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.634 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj9 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:c7J+PFQGdMBezT1F/ex4ZcWYYfE= 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c7J+PFQGdMBezT1F/ex4ZcWYYfE='} 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e2f3efd86e4a072-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031ba164699313a2e-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj10 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj10 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj10 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.637 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj10 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uwkUcDjPftgAmWk6bT1daH43lqs= 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uwkUcDjPftgAmWk6bT1daH43lqs='} 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000009c0e787083d4dad-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a06582044ef7dd5-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj11 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj11 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj11 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.640 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj11 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:PVheb51mJkOSvxS6QA+Qy2mdG5I= 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PVheb51mJkOSvxS6QA+Qy2mdG5I='} 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022508b49228f4498-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a33cf1325f0d0314-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj12 2024-04-10T00:01:36.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj12 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj12 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.643 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj12 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wg6bp56vHI0k/IMQH6Y1Ae9OUz4= 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wg6bp56vHI0k/IMQH6Y1Ae9OUz4='} 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f91ce26dca546f5-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecca3b6beee1363d-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj13 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj13 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj13 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj13 2024-04-10T00:01:36.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:juui1wrwD/LiPlnrjZsN+EMxZJ0= 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:juui1wrwD/LiPlnrjZsN+EMxZJ0='} 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d693f823bec30fb-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001192c082aceec0e5-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj14 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj14 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj14 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj14 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:CLKH09adS73uWnHrQruxC8rQP+Q= 2024-04-10T00:01:36.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CLKH09adS73uWnHrQruxC8rQP+Q='} 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cc2e0e9896cdf25-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.650 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8edc2767f409ad0-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj15 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj15 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj15 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj15 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nud1lICN0zogbFf1xeaFv4Gtxqo= 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nud1lICN0zogbFf1xeaFv4Gtxqo='} 2024-04-10T00:01:36.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8e6366840670526-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.653 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea25a09c7014450c-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj16 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj16 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj16 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj16 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XhKUyj906dtFrv3xBtx/joY/14E= 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XhKUyj906dtFrv3xBtx/joY/14E='} 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000428895689ff0f48c-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.656 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5e760d5db72fc2b-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj17 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj17 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj17 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj17 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uq/+RAETIfBdXtZ6RAIBvo7M+/k= 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uq/+RAETIfBdXtZ6RAIBvo7M+/k='} 2024-04-10T00:01:36.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000eb0b6e9acaeacb6-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.659 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000469cb12e97f7105-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj18 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj18 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj18 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj18 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cStLKI83yRHN72/QXnt/tXqfv04= 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cStLKI83yRHN72/QXnt/tXqfv04='} 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cc7ee4f20bef73f-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2cd716ab06efc77-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj19 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj19 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj19 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.664 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj19 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:BTVEnPHfiAnSlODyz5xWBFzjb5k= 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BTVEnPHfiAnSlODyz5xWBFzjb5k='} 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ae012926e48c040-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008addd01b259e52c9-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj20 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj20 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj20 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.667 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj20 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:T3l7YQEedu1Y98RZjb2GgsGRiYg= 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T3l7YQEedu1Y98RZjb2GgsGRiYg='} 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd961b341d5390ed-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000257b41e9e85c33bc-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj21 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj21 2024-04-10T00:01:36.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj21 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj21 2024-04-10T00:01:36.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uVcPnDl5XE6S1X0PymqtqWFyrhg= 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uVcPnDl5XE6S1X0PymqtqWFyrhg='} 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002391d32d6737b9c9-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020ee9ad31e335f11-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj22 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj22 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj22 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj22 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.673 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Jn/dP+IrA4rTNp1oVZZZr43H5Xc= 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Jn/dP+IrA4rTNp1oVZZZr43H5Xc='} 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aae320511f3386e3-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.674 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbec4fb9bf1e07b7-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj23 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj23 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj23 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj23 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:J5Ih/Af4Rywj7wt24/jh8R2B+EA= 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J5Ih/Af4Rywj7wt24/jh8R2B+EA='} 2024-04-10T00:01:36.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008cdfd5bac3149373-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.677 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a42250d9814d60eb-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj24 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj24 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj24 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.679 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj24 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:HozymsBHn2VijBA6b/JEUyorRZw= 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HozymsBHn2VijBA6b/JEUyorRZw='} 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5c79178ab24c638-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.682 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000786847deaaeda736-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj25 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj25 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj25 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj25 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qG34/Gm3fs4/t5TVY5M2glOns2o= 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qG34/Gm3fs4/t5TVY5M2glOns2o='} 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052139afcc1fb9eeb-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7b5972e06dd06ec-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj26 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj26 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj26 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj26 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yPVqljTRRY7VS6uJ+/+e0gURjps= 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yPVqljTRRY7VS6uJ+/+e0gURjps='} 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4b333a86f3eeebf-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b79a4ab406d67d8c-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj27 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj27 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj27 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj27 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:di1nN7RnI8c4yI5Ndo2Z3IzDRUs= 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:di1nN7RnI8c4yI5Ndo2Z3IzDRUs='} 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f63b4dd205fac368-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa29b0d3a58d6c05-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj28 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj28 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj28 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.692 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj28 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SLaqBniVM5u053a/tIjOuAeR/Jo= 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SLaqBniVM5u053a/tIjOuAeR/Jo='} 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb1515e7fe516ddc-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d69994fa1fa46ae0-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj29 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj29 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj29 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.695 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj29 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Na8hESqpD+Lrv6vU0WH69V4vwxM= 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Na8hESqpD+Lrv6vU0WH69V4vwxM='} 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1adff8a0612d30b-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007fb11c118a911e6d-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj30 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj30 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj30 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.698 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj30 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XVvaLWNZ7fFInV5KZeoUOyayyp8= 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XVvaLWNZ7fFInV5KZeoUOyayyp8='} 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff3572dd9f3ea616-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dacb105c28158127-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj31 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj31 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj31 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.701 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj31 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:n5Z4xtG1SeNMzFoV3QxAkxBErAg= 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n5Z4xtG1SeNMzFoV3QxAkxBErAg='} 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6961af09730aa16-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f24734ba78aa028-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj32 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj32 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj32 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.704 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj32 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6qvAlh+lTUs1Ikz3ch2MnHHF8RI= 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6qvAlh+lTUs1Ikz3ch2MnHHF8RI='} 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf71df5c1c39b7cd-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8d6be22f1623d9d-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj33 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj33 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj33 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.707 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj33 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hlJwxawneth5PJApWfevcfiVZ1E= 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hlJwxawneth5PJApWfevcfiVZ1E='} 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007928afb4d7ae2dd7-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004fe107373dc22a8-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj34 2024-04-10T00:01:36.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj34 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj34 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.710 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj34 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:I+gBbHDNdINxzgKTZ+C7OZuu4ZM= 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I+gBbHDNdINxzgKTZ+C7OZuu4ZM='} 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f732ec28839af47-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fa1285f5c3f4eab-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj35 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj35 2024-04-10T00:01:36.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj35 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj35 2024-04-10T00:01:36.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Kt7/kGsSwacCHEzvqKVMupO7JXw= 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kt7/kGsSwacCHEzvqKVMupO7JXw='} 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000746d0639f1b53a6e-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c05e198d4249df90-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj36 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj36 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj36 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj36 2024-04-10T00:01:36.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mtbcm+g/xHMJytbPNqIq9YgAl3o= 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mtbcm+g/xHMJytbPNqIq9YgAl3o='} 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c51198a1f8345f75-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be198d5886b9f543-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj37 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj37 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj37 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj37 2024-04-10T00:01:36.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:g3TF7lG1CbtnuFX3nXtMqwWw8Zo= 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:g3TF7lG1CbtnuFX3nXtMqwWw8Zo='} 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aff201b8b2d8aa7e-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba75d984b462bbe5-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj38 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj38 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj38 2024-04-10T00:01:36.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj38 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.722 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+QyNQvcXQzXbZAC0XyoPdMyycUw= 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+QyNQvcXQzXbZAC0XyoPdMyycUw='} 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a071bc8a145dcfa7-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f43fbacd3706f493-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj39 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj39 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj39 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj39 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vYRXWpsWm/mH2tRVPaxQSTwT328= 2024-04-10T00:01:36.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vYRXWpsWm/mH2tRVPaxQSTwT328='} 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ff8541030cedbd7-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.726 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000986a9aa99bf7d960-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj40 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj40 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj40 2024-04-10T00:01:36.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj40 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Q1Iw26zfxaefIIj6oncSmG+QmzA= 2024-04-10T00:01:36.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q1Iw26zfxaefIIj6oncSmG+QmzA='} 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e647ba874532be5d-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.729 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f1bf3ac8bc7c1a2-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj41 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj41 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj41 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj41 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:l67q8B5UiIc7mIgCI1vCfj3Ejb0= 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:l67q8B5UiIc7mIgCI1vCfj3Ejb0='} 2024-04-10T00:01:36.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f5bf76cc033c554-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.732 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU= 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5SXC7GFt2da4Yoiobl5xWh67jUU=', 'Content-Length': '0'} 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000960f758fc9a31117-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj42 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj42 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj42 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:03 GMT 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj42 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iiIxdEW+p0qlATdDGBTy0ifQzjY= 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:03 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iiIxdEW+p0qlATdDGBTy0ifQzjY='} 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a49d3c240b4cedf-006615b793-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.735 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b629878ae8601738-006615b793-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj43 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj43 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj43 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj43 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kWzkRoFVcU/KpJzaqXD9JxEyE5Y= 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kWzkRoFVcU/KpJzaqXD9JxEyE5Y='} 2024-04-10T00:01:36.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000963c3457bc328f15-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.738 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7ac1068243e647-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj44 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj44 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj44 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj44 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:M7F1DwLoED2H5cnOQuhzFJ6oOI0= 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M7F1DwLoED2H5cnOQuhzFJ6oOI0='} 2024-04-10T00:01:36.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000486737cb3f64528a-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.741 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006008949c268873d7-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj45 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj45 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj45 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj45 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0QrtXNGUlPNy/r9WZ3ahNfIuyt0= 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0QrtXNGUlPNy/r9WZ3ahNfIuyt0='} 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009610bf84f2dad0ea-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.744 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e0cc76a81bd5fd48-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj46 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj46 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj46 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj46 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:eJ1bP9mdQpCRfRrchINh+FMpkfs= 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eJ1bP9mdQpCRfRrchINh+FMpkfs='} 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f961d04f23e547e-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.747 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3c38108a18dff3f-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj47 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj47 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj47 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj47 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:T8tSR4cESz3YH/nPBMWM8Xi0q6k= 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T8tSR4cESz3YH/nPBMWM8Xi0q6k='} 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ede282c02c65d1a4-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.750 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9a28c1e36b39c74-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj48 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj48 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj48 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.751 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj48 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5WA3rl4QkO+gbwWFo4J3zQ1Es5I= 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5WA3rl4QkO+gbwWFo4J3zQ1Es5I='} 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000211920d2ed0fd997-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/ 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/ 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/ 2024-04-10T00:01:36.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:/xryqqk-12/ 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM= 2024-04-10T00:01:36.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:oyMrSBq8sYxSagbZWV/r6BXK2yM=', 'Content-Length': '0'} 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000674f3a5de7c6ab62-006615b794-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-12/obj49 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-12/obj49 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-12/obj49 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:36.754 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:48:04 GMT 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:/xryqqk-12/obj49 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:I6noa5m+z9V/d1+M0U6ZDTcCL+U= 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 21:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:I6noa5m+z9V/d1+M0U6ZDTcCL+U='} 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d61c1b24c60733ce-006615b794-1112-a2'), ('Date', 'Tue, 09 Apr 2024 21:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:36.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699281.708318_73.1", 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:48:01.708318Z" 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:04.354+0000 7fe789f86780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:04.355+0000 7fe789f86780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.343778_63.1", 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.343778Z", 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T21:48:04Z" 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:36.762 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:36.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:36.764 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.343778_63.1", 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.343778Z", 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.765 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.766 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T21:48:04Z" 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699220.343778_63.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.343778Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699220.343778_63.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712699281.708318_73.1 target=1_1712699220.343778_63.1 2024-04-10T00:01:36.767 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:09.467+0000 7fe6a0534780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:09.468+0000 7fe6a0534780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:36.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699281.708318_73.1", 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:48:01.708318Z", 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:36.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T21:48:09Z" 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.773 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.774 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.775 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699281.708318_73.1", 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:48:01.708318Z", 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:36.776 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:36.777 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T21:48:09Z" 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712699281.708318_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:48:01.708318Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712699281.708318_73.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:36.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:09.583+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:14.584+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:19.585+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:24.584+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:29.585+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:34.586+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:39.586+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:44.587+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:49.587+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:54.588+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:48:59.589+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:04.589+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:09.590+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:14.591+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:19.591+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:24.592+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:29.593+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:34.593+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:39.594+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:44.595+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:49.595+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:54.597+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:49:59.598+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:04.597+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:09.598+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:14.599+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:19.599+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:24.600+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:29.601+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:34.601+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:39.602+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:44.602+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:49.603+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:54.604+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:50:59.604+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:04.605+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:09.606+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:14.607+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.781 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:19.607+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:24.608+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:29.609+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:34.610+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:39.611+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:44.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:49.611+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:54.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:51:59.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:04.612+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:09.613+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:14.614+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:19.614+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:24.615+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:29.616+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:34.616+0000 7fe38fc07780 1 waiting to reach incremental sync.. 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:39.618+0000 7fe38fc07780 1 bucket sync caught up with source: 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: local status: [00000000007.24.6, 00000000004.12.6, 00000000004.14.6, 00000000004.13.6, 00000000006.13.6, 00000000005.20.6, 00000000004.19.6, 00000000005.14.6, 00000000002.7.6, 00000000004.11.6, 00000000004.17.6] 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: remote markers: [00000000007.24.6, 00000000004.12.6, 00000000004.14.6, 00000000004.13.6, 00000000006.13.6, 00000000005.20.6, 00000000004.19.6, 00000000005.14.6, 00000000002.7.6, 00000000004.11.6, 00000000004.17.6] 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:39.618+0000 7fe38fc07780 0 bucket checkpoint complete 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=xryqqk-11 zones={a2, a1} 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM= 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM=', 'Content-Length': '0'} 2024-04-10T00:01:36.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036fcdb8d49719b1a-006615b8a7-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/ 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.785 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM= 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kXG2sDOyYdPv+Gjw4oJNJghL1jM=', 'Content-Length': '0'} 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '294'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070c0d2ba4394154f-006615b8a7-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.787 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:/xryqqk-11/?versions 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b071671708afbde-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj1 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj10 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj12 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj13 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj14 2024-04-10T00:01:36.788 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj15 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj16 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj18 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj2 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj20 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj21 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj24 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj25 2024-04-10T00:01:36.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj26 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj27 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj28 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj3 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj30 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj32 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj35 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj36 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2024-04-10T00:01:36.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj39 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj4 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj40 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj41 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj42 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj43 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj45 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj47 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj48 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj49 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj5 2024-04-10T00:01:36.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj6 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj7 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj8 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj9 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:/xryqqk-11/?versions 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab14618761cf4457-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj1 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj10 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj12 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj13 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj14 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj15 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj16 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj17 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj18 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj19 2024-04-10T00:01:36.794 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj2 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj20 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj21 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj23 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj24 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj25 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj26 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj27 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj28 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj29 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj3 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj30 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj31 2024-04-10T00:01:36.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj32 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj34 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj35 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj36 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj37 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj38 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj39 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj4 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj40 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj41 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj42 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj43 2024-04-10T00:01:36.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj45 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj46 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj47 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj48 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj49 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj5 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj6 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj7 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj8 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj9 2024-04-10T00:01:36.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.798 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:/xryqqk-11/?versions 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a08dd81300de12-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?versions&max-keys=999 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.800 INFO:tasks.rgw_multisite_tests:/xryqqk-11/?versions 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko= 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6OgkJ392zQSE1/6hJCN2wkxynko=', 'Content-Length': '0'} 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ba8b2917225c93b-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11999falseobj1nulltrue2024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj10nulltrue2024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj12nulltrue2024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj13nulltrue2024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj14nulltrue2024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj15nulltrue2024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj16nulltrue2024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj18nulltrue2024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj20nulltrue2024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj21nulltrue2024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj24nulltrue2024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj25nulltrue2024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj26nulltrue2024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj27nulltrue2024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj28nulltrue2024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj30nulltrue2024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj32nulltrue2024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj35nulltrue2024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj36nulltrue2024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj39nulltrue2024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj40nulltrue2024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj41nulltrue2024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj42nulltrue2024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj43nulltrue2024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj45nulltrue2024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj47nulltrue2024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj48nulltrue2024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj49nulltrue2024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj9nulltrue2024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj1 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000261bfdac56e7021a-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f01c93bc07dfb0c5-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.805 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg= 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg=', 'Content-Length': '0'} 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fff52879f097f9f2-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.807 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg= 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GwJmQm0ynWpFZeaBVcAD/n6sIZg=', 'Content-Length': '0'} 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e8064bb04a92ffd-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj1 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-10T00:01:36.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e2d93598b950d37-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1?versionId=null 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E= 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kLZzxD21EoBoYvLB8Obv/2xH64E=', 'Content-Length': '0'} 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aad22a58ec773d2-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.812 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd683c5937065fe8-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000900d30586391af9-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko= 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko=', 'Content-Length': '0'} 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f375078042aeb881-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.817 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko= 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JjTATCj5YThgWPcX3LC/Xq2S3Ko=', 'Content-Length': '0'} 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3a0f62e0a841cdc-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj1 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.819 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE= 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE=', 'Content-Length': '0'} 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e35cbbb42d04486-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj1 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj1 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj1 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj1 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE= 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4y8bWPoTajpzHmz0QVbw2YJR2gE=', 'Content-Length': '0'} 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e38579db120a553-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.822 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecac3bb40c5a22ee-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj1 2024-04-10T00:01:36.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a554e00439430e6-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj11000falseobj12024-04-09T21:48:01.725Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj10 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb959997e6da8f40-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.827 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e8946ef32fc87b5-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM= 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM=', 'Content-Length': '0'} 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c51b7c800aaad7ab-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM= 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0RVDDZ8/PQC02uSbbqSAguYHgM=', 'Content-Length': '0'} 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eaa45d75ef5a28ed-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj10 2024-04-10T00:01:36.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.832 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000427003c4064fad66-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.834 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10?versionId=null 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I= 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iJKEEv16pY0QsOv6vLJrFEex97I=', 'Content-Length': '0'} 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a48291995e421728-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081622a2aa1f746a-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.837 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004972b9a9702cb96e-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10 2024-04-10T00:01:36.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM= 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM=', 'Content-Length': '0'} 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f939583462ba2e73-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM= 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VYSaZcGTcPS/pwfJeIzIBM1yaOM=', 'Content-Length': '0'} 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c69767a0e317f8-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj10 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.842 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ= 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073ff9e68d5f54494-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj10 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj10 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj10 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj10 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ= 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Sf2ybjfcKwlPknuwJJ1jxpg0KWQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007243b43d2ccf0e14-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.845 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000588e653cf6f8dc10-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj10 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.847 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b61feb16fa9c7ba-006615b8a7-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj101000falseobj102024-04-09T21:48:01.967Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-10T00:01:36.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0a4802e3d286044-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000589dfde47698b30b-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.852 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA= 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA=', 'Content-Length': '0'} 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcc8f7fa68ffacbe-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA= 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1M2oY3Hi/b6KY/PSgxNiaz7lMCA=', 'Content-Length': '0'} 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1e44a993e2de720-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc0059f28623e5a0-006615b8a7-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.857 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11?versionId=null 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM= 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h56Nxme0wbyMzbiEG4MQGgbwotM=', 'Content-Length': '0'} 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049075f72e1fac1bf-006615b8a7-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:39 GMT 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A= 2024-04-10T00:01:36.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+JsPndrs0Zkdyhvq7mgMUqcLn+A=', 'Content-Length': '0'} 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c48b041ff34eeb57-006615b8a7-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfefbb2468133c3d-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.862 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw= 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw=', 'Content-Length': '0'} 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002339967d26a81e8f-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw= 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AXoG+GKmjFHrcHfthFhH6MUzVTw=', 'Content-Length': '0'} 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014632a78a36646e4-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.864 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.865 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o= 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o=', 'Content-Length': '0'} 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdc09791e73f5c9f-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj11 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj11 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj11 2024-04-10T00:01:36.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj11 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o= 2024-04-10T00:01:36.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:88jYwrbsD1p2luDEYXPLcOt2t3o=', 'Content-Length': '0'} 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d53a1f9861040d2-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d56a248d017aa70-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj11 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.870 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd50c23e0cc1f78a-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj111000falseobj112024-04-09T21:48:02.017Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj12 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.872 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008154156de9478a59-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.874 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d18179685090f918-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk= 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk=', 'Content-Length': '0'} 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073b255d4b37a8d7d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.877 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk= 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WylIPqUpsXOWAiEmn08sVf1Cqmk=', 'Content-Length': '0'} 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b717cd859b0f8215-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj12 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.879 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ef030979a4c97cb-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12?versionId=null 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI= 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F137RWtDlR3jiM5l75wf9/M22EI=', 'Content-Length': '0'} 2024-04-10T00:01:36.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce3f22a86cffa240-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005313c35820addb3f-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063a5e46db1d9be9b-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.887 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg= 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg=', 'Content-Length': '0'} 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ddd6b284fd70314-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg= 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EBIo/pIMlg6MINZ/Gq6vDIUvmPg=', 'Content-Length': '0'} 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d415ae3a667b409-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.889 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj12 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.890 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ= 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acf4bee53c711bca-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj12 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj12 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj12 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj12 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ= 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:LjoJMSCvWUiYKnQdOsMxshCsZRQ=', 'Content-Length': '0'} 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8d85d956671fca1-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.893 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d80ff5e69f77c51-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj12 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.895 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b2357bf6e34857-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj121000falseobj122024-04-09T21:48:02.026Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj13 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.897 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0dd68974c8832d7-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e330135cc5d1a968-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.900 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk= 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk=', 'Content-Length': '0'} 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096c51ee1f3caf985-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk= 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KxaS0toUgXB6WR+tFOVWE57C6jk=', 'Content-Length': '0'} 2024-04-10T00:01:36.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006032a59e39656496-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj13 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.903 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ecf076198a00c0d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13?versionId=null 2024-04-10T00:01:36.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c= 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i++7WGpuH057xLp+w6/vfiRYs3c=', 'Content-Length': '0'} 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c93d72be9d815ae-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097e415909ef9949c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a48022451e95699-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.910 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4= 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4=', 'Content-Length': '0'} 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002366d086586d8057-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4= 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pombj8dnPYknlSetljnhKwrm4+4=', 'Content-Length': '0'} 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b249ebb3dcfe0219-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj13 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13 2024-04-10T00:01:36.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY= 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY=', 'Content-Length': '0'} 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005229187ec76b6c95-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj13 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj13 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj13 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj13 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY= 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mvZeJJTf7XXiTdS+z1IbvcZ+qtY=', 'Content-Length': '0'} 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0fb002419a33921-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.916 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548edc7b0c2009e2-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj13 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000024834891bc18e97-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj131000falseobj132024-04-09T21:48:02.034Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj14 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d24d86d2f66a5a32-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.921 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f153d5682873b1bc-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.923 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU= 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU=', 'Content-Length': '0'} 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5c4e1eaca875691-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU= 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MoVwpuPQyqOy7ibZEcRsF3/GznU=', 'Content-Length': '0'} 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018784bb2a182d57e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj14 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.926 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000deaaf179d6feb027-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14?versionId=null 2024-04-10T00:01:36.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU= 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7XNYP+FJBuOPh9FYf5snjgo6QU=', 'Content-Length': '0'} 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a368af17ce02808-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e052b589aad18e7a-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.931 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ca507c53ab0bbcf-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14 2024-04-10T00:01:36.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo= 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo=', 'Content-Length': '0'} 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a166e64c4fb5dab-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo= 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kQY0ZA/Jx9U5CuMoLkw5pAlbdJo=', 'Content-Length': '0'} 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000977ff40dc4e21cfd-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj14 2024-04-10T00:01:36.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.936 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4= 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4=', 'Content-Length': '0'} 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93f983177471432-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj14 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj14 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj14 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj14 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4= 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tsL5Tst6Kn2DAmg/ZbJojhGYTn4=', 'Content-Length': '0'} 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c23a691d88afd2be-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.939 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b31f9ef8f587d55-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj14 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091097cc8479b0bc5-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj141000falseobj142024-04-09T21:48:02.089Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj15 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf816f93b7862078-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.944 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea7b164f4402a27c-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y= 2024-04-10T00:01:36.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y=', 'Content-Length': '0'} 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6e827b28b6f9015-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y= 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aEUgxdMeSj1ad6e+bjpm518aq8Y=', 'Content-Length': '0'} 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc262fec6d92ae0b-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj15 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.949 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd509f4abad4e02e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15?versionId=null 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU= 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mp0azd1sUmBc58/2cWFVEetJRtU=', 'Content-Length': '0'} 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c647ff5628e864-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.952 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ef52b96e86941c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005223fcfa28dc691d-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs= 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs=', 'Content-Length': '0'} 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048087bbd29c6cc1d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.957 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs= 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:B5c49GuqqLecClqkNcVPJCwWcDs=', 'Content-Length': '0'} 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000399dddb77017bbf0-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj15 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic= 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic=', 'Content-Length': '0'} 2024-04-10T00:01:36.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f6d5692c8e903d9-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj15 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj15 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj15 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.960 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj15 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic= 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ymqZiw4hj5Nz4JY1/h9vYxZMjic=', 'Content-Length': '0'} 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0057df4b10d47ea-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.962 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1aa45177c96c950-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj15 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008636d57fdb1c4a90-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj151000falseobj152024-04-09T21:48:02.097Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj16 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4f84ec8e01b6e83-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.967 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fd86e8fb5246a69-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk= 2024-04-10T00:01:36.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk=', 'Content-Length': '0'} 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d946ba06ffa6ca61-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk= 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sZPlVY0ezgZTboeMNL+DPPudOPk=', 'Content-Length': '0'} 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001274a5980674fa84-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj16 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.972 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e140812f1ee7c2e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16?versionId=null 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18= 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZbFdjHAL4GK5hD8cOcv6alN7m18=', 'Content-Length': '0'} 2024-04-10T00:01:36.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ac3e8917ca40ba9-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb077f62e65e4f3e-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.977 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54aec7ffeb14845-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU= 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU=', 'Content-Length': '0'} 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a5a58a6997bd46d-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.980 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU= 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TPQmfSwnx1tJZpl1WThnyOUYcVU=', 'Content-Length': '0'} 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f59cfb36d3f190f8-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj16 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM= 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM=', 'Content-Length': '0'} 2024-04-10T00:01:36.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b350f6fc43d8667-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj16 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj16 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj16 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.983 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj16 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM= 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:45lcH3AEShzbXRhl4LujYxBW/ZM=', 'Content-Length': '0'} 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e98ce11499f4910-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.985 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022aa14b50dc3422c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj16 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002610c14113bcca21-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj161000falseobj162024-04-09T21:48:02.144Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057865ff98ff5cafe-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.990 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002aa1717b6fbdaf39-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA= 2024-04-10T00:01:36.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA=', 'Content-Length': '0'} 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000673e479936a5af5e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA= 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3hw1SZx6EaOfkd5KZO1GFO8sRpA=', 'Content-Length': '0'} 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034113c91defaf0e8-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.995 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ca8bb9600c6572e-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17?versionId=null 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q= 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mCVlbNlffGJ095giMy14ym3Td3Q=', 'Content-Length': '0'} 2024-04-10T00:01:36.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030ec3068010beb8a-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj17 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:36.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ac1d8ab2dea2a3c-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:36.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.000 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b460ea41bc667b-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w= 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w=', 'Content-Length': '0'} 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe027e6e336eee20-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.003 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w= 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bkYNG8dgrOM6la6unvmn4vJj18w=', 'Content-Length': '0'} 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba3b4c112d13bdc6-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj17 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds= 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds=', 'Content-Length': '0'} 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c6cc276bef32825-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj17 2024-04-10T00:01:37.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj17 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj17 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.006 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj17 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds= 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0ICed8hjO6HbMUzfaSogSD+10Ds=', 'Content-Length': '0'} 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2538920ab53f896-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3626d59786bd1af-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj17 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.009 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a08d7cb0cfb70e64-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj171000falseobj172024-04-09T21:48:02.154Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj18 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.011 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dba64bc996ddb82f-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019cdee40d6727de8-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.014 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU= 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU=', 'Content-Length': '0'} 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b0a50a6c040d0999-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU= 2024-04-10T00:01:37.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QV6YrrthMHbenereoB8lB5VeRkU=', 'Content-Length': '0'} 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048380b1a0de2d430-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj18 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab91b8ea29c1eed0-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.019 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18?versionId=null 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM= 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nibvwBDgpRQ3mpTFOsb4SNUaidM=', 'Content-Length': '0'} 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bcf85b13b67de94-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.021 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc415627da04629-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f8c88d608b6a05-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.024 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI= 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI=', 'Content-Length': '0'} 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004f4cb35b2bf64b2-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI= 2024-04-10T00:01:37.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Im6nZVbW3Gr/pxMNHNFa13l7eJI=', 'Content-Length': '0'} 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6e51096f5b6c907-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj18 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.027 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk= 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk=', 'Content-Length': '0'} 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095755bee41fa034a-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj18 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj18 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj18 2024-04-10T00:01:37.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj18 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.029 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk= 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:s+ZiD4rT80yCmzeUYYsejxnNiUk=', 'Content-Length': '0'} 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000002c03a50a968513d-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000776709abbdbd9270-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj18 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.032 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066785bdbb5cd914c-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj181000falseobj182024-04-09T21:48:02.200Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.034 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000087025e1d3cda1fc-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-10T00:01:37.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c2de69d02495bb6-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o= 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o=', 'Content-Length': '0'} 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca3b7b85e56aa6a9-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.039 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o= 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JPApULkLuUlSc10PNQ703GMLT9o=', 'Content-Length': '0'} 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001654feff730ecce9-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-10T00:01:37.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2b40b0c69b5adab-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.042 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19?versionId=null 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8= 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pgIbtv23G/Sgzb5fuzecx+q/dH8=', 'Content-Length': '0'} 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006800cf467a4eb0cd-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.044 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069e785dd966ebf66-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c050642cba8872b-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.047 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ= 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb9d161b9dbc8c52-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ= 2024-04-10T00:01:37.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EOxkYQVmoRUNjoa0i/hPxcW6/EQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e1c636d3b1833e0-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj19 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8= 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8=', 'Content-Length': '0'} 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e427d0e4537f8f9-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj19 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj19 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj19 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj19 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.052 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8= 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u+Fvh7JcRo9nUvXBUnzSishrRs8=', 'Content-Length': '0'} 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fff979bb276bfb66-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e66bfd5ddde56b1-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj19 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.055 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc31e23d6d34b319-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj191000falseobj192024-04-09T21:48:02.209Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj2 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.057 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000096784bd98d989f41-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-10T00:01:37.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000463cfb3e0232a890-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.060 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8= 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8=', 'Content-Length': '0'} 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000873515efbf943bf5-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.062 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8= 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7ZAeohHSn+6JEIFwE9GkOHYfiB8=', 'Content-Length': '0'} 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2d899746930f5e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj2 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-10T00:01:37.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c7f912f642c9b4b-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2?versionId=null 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE= 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y7lI/RWw//q0v3qvhLF0IPr8LxE=', 'Content-Length': '0'} 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093395228e39f1d68-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.067 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052af4de31b890806-006615b8a8-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.069 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk= 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:F/uE13J5tHLB/x+DQTrXbBwMwWk=', 'Content-Length': '0'} 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd94ea1700d83f4c-006615b8a8-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk= 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk=', 'Content-Length': '0'} 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8679fe634c9a469-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.072 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk= 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aKFwSntMXP8KH9M3Kh456daXBZk=', 'Content-Length': '0'} 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000445cdae5c6dc038e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj2 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo= 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo=', 'Content-Length': '0'} 2024-04-10T00:01:37.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b04402f3c139c10-006615b8a8-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj2 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj2 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj2 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.075 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:40 GMT 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj2 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo= 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TZMLj940nSGIp4HN9seg0A684Wo=', 'Content-Length': '0'} 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001cdb091c6cfc20e-006615b8a8-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07f46eaa6174183-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj2 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbbb23f907839eba-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj21000falseobj22024-04-09T21:48:01.773Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj20 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.080 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000607faf49bb9cf2f8-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001368836f177820a4-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4= 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4=', 'Content-Length': '0'} 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000dee63fb8412e923-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.085 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4= 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4bNsBs9myUVOIncBNdi8dBR/cV4=', 'Content-Length': '0'} 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005be374ad7475e780-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj20 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.087 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fdc1f7159b9943b-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20?versionId=null 2024-04-10T00:01:37.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ= 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CNpbveNHPgY4mTG5XT346PpILSQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ee1829b3306694f-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000448c4837198251ce-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.092 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000822447a9af2ac487-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20 2024-04-10T00:01:37.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.094 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU= 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU=', 'Content-Length': '0'} 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000adebeb49ff9ca79-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU= 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Yr41Uca1Vjcg3mKcdn4pkb4aLuU=', 'Content-Length': '0'} 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a2750998d9e321f-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj20 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.097 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs= 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs=', 'Content-Length': '0'} 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052ba96dc6523ccb1-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj20 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj20 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj20 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj20 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs= 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e/9gz3DuOZeczYZtm6z/J54NJJs=', 'Content-Length': '0'} 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4b920a1f93a21fc-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005331f99a9f2d9217-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj20 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8bfb8036298bc63-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj201000falseobj202024-04-09T21:48:02.217Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj21 2024-04-10T00:01:37.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.104 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb55a5087a38e1e5-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-10T00:01:37.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000951f5b7a26e38c-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4= 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4=', 'Content-Length': '0'} 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e198d3006a322de-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.109 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4= 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5RCFMPdQJKaF432k/WTGK+wJnr4=', 'Content-Length': '0'} 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d2c10c4db51f71b-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj21 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.111 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9bfd90b7d88ff41-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21?versionId=null 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY= 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zYVl/Fv2PKhT5B8ibhtqMi42vGY=', 'Content-Length': '0'} 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000600be81b5b389390-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.114 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068a81e29c809a816-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.116 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e99265b170f82f-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU= 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU=', 'Content-Length': '0'} 2024-04-10T00:01:37.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a405699c2e9e1086-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.119 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU= 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k/ev6p439CTWK3PoHWtz5iFCKrU=', 'Content-Length': '0'} 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c692bb8a7c6bdae-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj21 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc= 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc=', 'Content-Length': '0'} 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da1ff16465c5a4c7-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj21 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj21 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj21 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.122 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj21 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc= 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wAmGc9GUE40b/Yg9rO2JAfL2lKc=', 'Content-Length': '0'} 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001123da2acd0164a9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6f53dd9c1a3b88-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj21 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fc55fa8ef02ed27-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj211000falseobj212024-04-09T21:48:02.266Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.128 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055c59dfe6afa45a4-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df694db0f7357dd0-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.131 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg= 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg=', 'Content-Length': '0'} 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cddc42c737e2bae-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg= 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hugeiAX38b9BtR78U8UueeMRlqg=', 'Content-Length': '0'} 2024-04-10T00:01:37.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004299e4af80833424-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d87246f08b590186-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.136 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22?versionId=null 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ= 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bb7O7ZrXXlcgC2/DxjnKZyw6miQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a88b740814d53c9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.138 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000578e2c898fe49ca6-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ace22c22b5f927fe-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM= 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM=', 'Content-Length': '0'} 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009282b1583a89eee8-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.143 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM= 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wNGbYUspI6aPx3YGiP7re/FMmSM=', 'Content-Length': '0'} 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000317f16e3d773fff9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10= 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10=', 'Content-Length': '0'} 2024-04-10T00:01:37.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000374cc55c45301e42-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj22 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj22 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj22 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.146 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj22 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10= 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:H71KSm4TmVuRsFgKENxudR45F10=', 'Content-Length': '0'} 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7df85f6c82d8c51-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.148 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006943ad6c0f50196f-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj22 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059e5066b2e7e4061-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj221000falseobj222024-04-09T21:48:02.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041fe046e8fb31dba-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.153 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f83a5f379fafb74-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw= 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw=', 'Content-Length': '0'} 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003522b9a7210ad912-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.156 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw= 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UYYAXhO7cEXGeWnrdEW8ayRX1uw=', 'Content-Length': '0'} 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001990df138607d202-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bda72cda5e499662-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23?versionId=null 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI= 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y2fSi2NL4Xc34VpvkRUpk7pX8uI=', 'Content-Length': '0'} 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a86754cd8e6a8e4-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.161 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8da02ac1e66283a-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.163 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4754bfacfcc7fe9-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY= 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY=', 'Content-Length': '0'} 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd9ddf2c98d959b2-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.166 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY= 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E2Cre4cm+xupAG5MxzmZf+AOdEY=', 'Content-Length': '0'} 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c18b6f184c2abbf3-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj23 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk= 2024-04-10T00:01:37.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk=', 'Content-Length': '0'} 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d0895b5c919d53a-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj23 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj23 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj23 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.169 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj23 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk= 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WuHBPHxnzw3VPN8qT6HC7wmpmvk=', 'Content-Length': '0'} 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc3b80ad9c0faeb0-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a84fd33b002cf9e3-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj23 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004460b2cdbf7424a2-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj231000falseobj232024-04-09T21:48:02.324Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj24 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053089172e2e94002-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.176 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3790ca7d27b6c70-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0= 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0=', 'Content-Length': '0'} 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d34c6e2f3080639-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0= 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yVaIHwRzXuDDj3aHhxCRPxOsCN0=', 'Content-Length': '0'} 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000208367efa42c3e60-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj24 2024-04-10T00:01:37.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.181 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc99a8b5e5c35960-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24?versionId=null 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM= 2024-04-10T00:01:37.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLBBPDtdrPhPnt2wNcdUyrMIlCM=', 'Content-Length': '0'} 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b0103803316d687-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.184 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089785763d18385cf-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c85f497f2e2a99ba-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk= 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk=', 'Content-Length': '0'} 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095dce07e58bc0554-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.189 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk= 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WeAWlukMi164w0xCLo4+DGk7Qyk=', 'Content-Length': '0'} 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c777546bb8b4c6a9-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj24 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE= 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE=', 'Content-Length': '0'} 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e034a2623e52ce70-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj24 2024-04-10T00:01:37.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj24 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj24 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.192 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj24 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE= 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SIvG3tN2Iuir4DRDR1QZfGm9jFE=', 'Content-Length': '0'} 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077cd6d4bc0513f19-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058ee2a131461444e-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj24 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dd2943ae0b96a4b-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj241000falseobj242024-04-09T21:48:02.332Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj25 2024-04-10T00:01:37.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.197 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000810d92113fad3bd4-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d75f2ba5f500677-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q= 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q=', 'Content-Length': '0'} 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfdd7a836bba1a37-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.202 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q= 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vkg+JGUDB1XYp/xk1Hc07kI5i4Q=', 'Content-Length': '0'} 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054399c7653b93abe-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj25 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-10T00:01:37.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f74aea9fd1bb820b-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25?versionId=null 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw= 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JiXGRgh8m5ctp+eKcRb8Yn0knWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004741981ca9cddd6c-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.207 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9d96706f89328d8-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e019626fc4bbcb3-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE= 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE=', 'Content-Length': '0'} 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000059d47a49b202a75-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.212 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE= 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FnuyDAUaiMLzS2OkV1EpiM8CNlE=', 'Content-Length': '0'} 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f234521ac9275761-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj25 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g= 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g=', 'Content-Length': '0'} 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ef6fd78771ba865-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj25 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj25 2024-04-10T00:01:37.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj25 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj25 2024-04-10T00:01:37.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g= 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p2nuSuT4momy5hVeyVFQxcAT/+g=', 'Content-Length': '0'} 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2351ee9922f1f91-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048de3531c5750625-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj25 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000931d1ca0ec130bfb-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj251000falseobj252024-04-09T21:48:02.340Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj26 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.220 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab94dae7c50e7a9a-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-10T00:01:37.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044547071f5f423a3-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.223 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo= 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo=', 'Content-Length': '0'} 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f5d98c4563b07b6-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo= 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Wugdrg/lF6bo2tQVNDAiGRaDQmo=', 'Content-Length': '0'} 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000216eb56550d3c3c8-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.225 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj26 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005f6d270570868091-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.228 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26?versionId=null 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY= 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TKj63GnNIjVuG/tc/1VUGLGcGTY=', 'Content-Length': '0'} 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f450fb4a101f41c-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003828734feeb40206-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b58be1a70ef32eb-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.233 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg= 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg=', 'Content-Length': '0'} 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b2201f21ed4b99d-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg= 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:szUXA7dHXjl1JGN5Sbpb3VsZcPg=', 'Content-Length': '0'} 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf9271a9f0823e8b-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj26 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.236 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8= 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8=', 'Content-Length': '0'} 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e124ef327b177386-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj26 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj26 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj26 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj26 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8= 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RINzoiTrllZR7vAnfj4JiNXp3w8=', 'Content-Length': '0'} 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007385bc8ed42ac28-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.239 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a59097dcdb51c7-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj26 2024-04-10T00:01:37.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cf26ac06ab5c9ac-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj261000falseobj262024-04-09T21:48:02.389Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj27 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049ae5aa904cc0206-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.244 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015618b491878f5ae-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM= 2024-04-10T00:01:37.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM=', 'Content-Length': '0'} 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d608077dfa4f7a8-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM= 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xdVzgkUWJC0VvXoZbD3mvrX5vnM=', 'Content-Length': '0'} 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fba2fdd9250e2c7-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj27 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.249 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046d8ffaf0591c4da-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27?versionId=null 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o= 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JCS/lDyrcYjDPkIJNN/D5CAdP6o=', 'Content-Length': '0'} 2024-04-10T00:01:37.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003161fab8699f98c8-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008693e3e2145edae9-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.254 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4bbc3f77391a654-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y= 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y=', 'Content-Length': '0'} 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f7ec84b7b24ea47-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.257 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y= 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:VFEslxnOtre/L8z26G650BStx0Y=', 'Content-Length': '0'} 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063af9af0ad0d8983-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj27 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg= 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg=', 'Content-Length': '0'} 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f61f4375249b271-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj27 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj27 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj27 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.260 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj27 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg= 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yy5n2yDTVP9FcodjsAo5fLF4tYg=', 'Content-Length': '0'} 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efd5bac6b25f5905-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9f6fc2da47441d5-006615b8a9-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj27 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic= 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ZXqMuUGLtiraQjKyLGizKE6iic=', 'Content-Length': '0'} 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028810939bd05c3f6-006615b8a9-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj271000falseobj272024-04-09T21:48:02.399Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj28 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.265 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4= 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4=', 'Content-Length': '0'} 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef3d08d1e32bca3c-006615b8a9-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:41 GMT 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.267 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4= 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:A6b8w9iGaxHerzyzUYlsaiAofL4=', 'Content-Length': '0'} 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009839b21786d1e8fa-006615b8a9-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4= 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4=', 'Content-Length': '0'} 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe98be0ec16c749f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.270 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4= 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E8qkmTsjMHiwSA7p3ZI8X3Rgmt4=', 'Content-Length': '0'} 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008fb8554a5ef025ff-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj28 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.272 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww= 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww=', 'Content-Length': '0'} 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb92510717c616da-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28?versionId=null 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww= 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/OeSxjxYAXdbqt7x5jheqQk4Fww=', 'Content-Length': '0'} 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c54ba3a8bcb29b6-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.275 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2188cfbab40cac7-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdff60cf9a6042cc-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw= 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw=', 'Content-Length': '0'} 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000781bd9547844d35e-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.280 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw= 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UNrCXZOl/i22RuHpuEsgCAdLihw=', 'Content-Length': '0'} 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003378258b5bf1f4ea-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj28 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk= 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk=', 'Content-Length': '0'} 2024-04-10T00:01:37.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c37cd9f365a1b23f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj28 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj28 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj28 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.283 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj28 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk= 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0cEosn+6wkDyea9/t0sWLnVx2tk=', 'Content-Length': '0'} 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000765a649177994833-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.285 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41de1123ad1fabb-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj28 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093c59ed3d768c866-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj281000falseobj282024-04-09T21:48:02.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.287 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004464579b1dd6f08d-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.290 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000457645289995050d-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc= 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc=', 'Content-Length': '0'} 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007193d9977d1a3cd-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.293 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc= 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gVHtUtA1jW7Umx8JN5Kxpckvpzc=', 'Content-Length': '0'} 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008592d659c02416ec-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000555e618a79336fb2-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29?versionId=null 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo= 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mhfo303Gi5VUjcphhtEx5555BDo=', 'Content-Length': '0'} 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b47794a65ba4f1ee-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.298 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb3d420df8850f4b-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.300 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138bbe9da8f007b9-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs= 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs=', 'Content-Length': '0'} 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be6a8a068f704b88-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.303 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs= 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w9UKMoB6/cGfK58jmYjTRVEztIs=', 'Content-Length': '0'} 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000570b919894da7f76-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj29 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs= 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs=', 'Content-Length': '0'} 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb8b6daa8f6ec058-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj29 2024-04-10T00:01:37.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj29 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj29 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.306 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj29 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs= 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bRm/5WtWj9+9a2pQo/CDIwqwsXs=', 'Content-Length': '0'} 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000188494966e5c51cd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4037aced0dc32ac-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj29 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.309 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d914a5f8d0e9d422-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj291000falseobj292024-04-09T21:48:02.416Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj3 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3613dd51cc2a3c1-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017c8a5450c926efd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.314 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg= 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg=', 'Content-Length': '0'} 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036214a5caeadd0b4-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg= 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:R5i+MyTsdgUPYCg/nwvWr9Z2rSg=', 'Content-Length': '0'} 2024-04-10T00:01:37.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000213bf504c9b6c62f-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj3 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.317 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027524190f554d28b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3?versionId=null 2024-04-10T00:01:37.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU= 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EwAk6y3lXju1UKzpEN91Wx5+RjU=', 'Content-Length': '0'} 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005856995e59851206-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1677678598afd8e-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.322 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbce69e769dee13f-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA= 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA=', 'Content-Length': '0'} 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acb3d0af6fc6df91-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.325 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA= 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yfGjNKLHyJnnH2hFjr/NNi84FlA=', 'Content-Length': '0'} 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acf845960b629df8-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj3 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8= 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8=', 'Content-Length': '0'} 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af17a84d37f8ce0b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj3 2024-04-10T00:01:37.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj3 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj3 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.328 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj3 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8= 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZEwnfTlQxftsrtE6FWJAeGgo0l8=', 'Content-Length': '0'} 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bab5071c9b10bfd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03dcfbd932fcdd6-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj3 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.331 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ed797f01d14196f-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj31000falseobj32024-04-09T21:48:01.783Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj30 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055d4131e551780f3-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.334 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dffddcb2a754dafd-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic= 2024-04-10T00:01:37.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic=', 'Content-Length': '0'} 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006dc68089cf297ebc-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic= 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vrPNMkWeOvZ6AaI82IXx8DpNLic=', 'Content-Length': '0'} 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b34a7353ec8c1760-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj30 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a12d3c243cdc80eb-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30?versionId=null 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E= 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MXds5oLDptH/d5JDd2VeG2c3g8E=', 'Content-Length': '0'} 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abda4e17631356c2-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.342 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1825f9be69ee84d-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c79efe941044544-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.345 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg= 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg=', 'Content-Length': '0'} 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070c8db4a62483ef7-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg= 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8xHA9BPNgUDAfbSfCeCof7Rdrpg=', 'Content-Length': '0'} 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005859f0a71026667e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.347 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj30 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.348 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY= 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY=', 'Content-Length': '0'} 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb05f6261cb1a7a8-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj30 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj30 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj30 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj30 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY= 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WLB6zl1EsgycWU3dKUnSXh9+oKY=', 'Content-Length': '0'} 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000744f15ed97169eb2-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044e2a2c162cd1c0a-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj30 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058bd0867e1f307ae-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj301000falseobj302024-04-09T21:48:02.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.354 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000330c2dbdfceeade-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007451981b606dca01-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.357 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY= 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY=', 'Content-Length': '0'} 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000512af2107049c41f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY= 2024-04-10T00:01:37.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4PbsT0QTdN3oIgA5RRgCo8W3ksY=', 'Content-Length': '0'} 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e9097a87368c9dc-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069051853f6edb83f-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.362 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31?versionId=null 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE= 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r0TNi8rm8jDwZTPOELNIAqp9HOE=', 'Content-Length': '0'} 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000192626bfcadbcef2-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000121a3a6ea77bc3a8-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001710d2cec6bd3625-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.367 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg= 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg=', 'Content-Length': '0'} 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000629e517905762560-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.368 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg= 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mm5asBHa9pSfFBfJcf7yTn9f/lg=', 'Content-Length': '0'} 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000829284b538454ccb-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj31 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go= 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go=', 'Content-Length': '0'} 2024-04-10T00:01:37.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000947871d58d138c0c-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj31 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj31 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj31 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.371 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj31 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go= 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lnOlxmq405eDt2Xof0kP4JiC7Go=', 'Content-Length': '0'} 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f2ba00ef5b0c08-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f549495567bd580-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj31 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.374 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3a23334ad1a143e-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj311000falseobj312024-04-09T21:48:02.474Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj32 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ffa179b985fb77c-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfee6f6bbdf3ef67-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.379 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM= 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM=', 'Content-Length': '0'} 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000966e48b6eb190fbe-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM= 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bEKRMNJHTWQDI/TUsYHI+tNOZkM=', 'Content-Length': '0'} 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e286148a17643c0-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj32 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.382 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6ce18a3ce31d41b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32?versionId=null 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg= 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GjYV5tpjFnjfT/TF+K7A45ioBXg=', 'Content-Length': '0'} 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f7b5f5fdba3fdf-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.385 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd53d7a5a19a1df8-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022b32250b257c3a8-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.388 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I= 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I=', 'Content-Length': '0'} 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9f27c5c95beb848-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I= 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RQi+ygOiXOyqqqT3dUK9ukRkV4I=', 'Content-Length': '0'} 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a4c2443c09152388-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj32 2024-04-10T00:01:37.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.391 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs= 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs=', 'Content-Length': '0'} 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5483e64bed333de-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj32 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj32 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj32 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.392 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj32 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs= 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Bx0NsXfaoT2d2TQ1C6T3Iz30YAs=', 'Content-Length': '0'} 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004686564320fa874d-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.394 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224eef154f29818e-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj32 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030e4d28339de9850-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj321000falseobj322024-04-09T21:48:02.482Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.396 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3f15539e261c478-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.399 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056ce783c3f3716a0-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s= 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s=', 'Content-Length': '0'} 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da797f84d3635c08-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.402 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s= 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Gn3wuMHNZ1x/ptYNFGu9PmvMw1s=', 'Content-Length': '0'} 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053eb32b41cf020b4-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.404 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8f62a6a4ee041bb-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33?versionId=null 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI= 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DV2YbKRE2ISJwSRBam+1bncUTcI=', 'Content-Length': '0'} 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f34e5c98ee0cdaee-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.407 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097d17f9416da040c-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032c5c50a61000be9-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.410 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o= 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o=', 'Content-Length': '0'} 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000800d889970d2161d-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o= 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/2Lm1XDA8atoVGinS2crAbhop0o=', 'Content-Length': '0'} 2024-04-10T00:01:37.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009aadbc309d87e35e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.413 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc= 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc=', 'Content-Length': '0'} 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7c40e1931e68dae-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj33 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj33 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj33 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj33 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc= 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Q/84jT22yE8/pIH1r+oYU5apQnc=', 'Content-Length': '0'} 2024-04-10T00:01:37.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab1309b03178998a-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.416 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c95e4beb788d34-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj33 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.418 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6128743fcc1fa5-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj331000falseobj332024-04-09T21:48:02.532Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-10T00:01:37.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000260169cb358c1ff5-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.421 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009afad587a7c8bbd3-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.423 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws= 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws=', 'Content-Length': '0'} 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003978c6dfeaaedaff-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws= 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UllhbG/7qlnz6fggfCHWe6Muxws=', 'Content-Length': '0'} 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000806d6cff1e6a4dfc-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.426 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000256097f3bbae3085-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34?versionId=null 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0= 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ffWDrQcfK7qwwTcYPQfg4oduvg0=', 'Content-Length': '0'} 2024-04-10T00:01:37.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1c591e7fcc8a899-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.429 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000083bcd68fb666572-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.430 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.431 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5cbac2c45fd5207-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.432 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc= 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc=', 'Content-Length': '0'} 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb9cf58b8d09310c-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.433 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34 2024-04-10T00:01:37.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc= 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ssb8QK4gmQfjZU+6SiAgG3+lEVc=', 'Content-Length': '0'} 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef4602b27da1e6aa-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj34 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4= 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4=', 'Content-Length': '0'} 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076ef3ecace4bc9eb-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj34 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj34 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj34 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj34 2024-04-10T00:01:37.437 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4= 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FEqEgXWE8b2LW893Hx1lCnQc2J4=', 'Content-Length': '0'} 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041c45357796727d9-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.438 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010505f8fd822a202-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj34 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.440 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed80050929d0fe5-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj341000falseobj342024-04-09T21:48:02.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj35 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.442 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000872d7282d4be376b-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003d11ae69b35d6a94-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.445 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug= 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug=', 'Content-Length': '0'} 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d71d183811d11553-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug= 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:p1HlXX8OzL17S3g2tViiO7/+7Ug=', 'Content-Length': '0'} 2024-04-10T00:01:37.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d121ba883636ea50-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj35 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007080dc45f5f939e0-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.450 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35?versionId=null 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c= 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3t9KBRwI9ns/dOMXU0CqpKal/+c=', 'Content-Length': '0'} 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099c802ae4d327b39-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f016cb4b1483a0-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.453 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d94d9c5b87d8b2-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.455 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s= 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s=', 'Content-Length': '0'} 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3a91844164f28d7-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s= 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JpaORvG87D7W0d/J7pamnk0pq/s=', 'Content-Length': '0'} 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004db24409ae59dd89-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj35 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.458 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc= 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc=', 'Content-Length': '0'} 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015ee93c0ec92d0ef-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj35 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj35 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj35 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj35 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc= 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8GfPsTB5LYoqo6y9s3M/5tYIlgc=', 'Content-Length': '0'} 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000031d31a5d6d84f2e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.461 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000706eb23c2535f4e9-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj35 2024-04-10T00:01:37.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.463 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b284f51823dce53d-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj351000falseobj352024-04-09T21:48:02.593Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj36 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa6c72a8e20972db-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.466 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000317ec9cf2febf8e6-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng= 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng=', 'Content-Length': '0'} 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031c77201f2feb0ef-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng= 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JoO3snSMyOvrLaC83BNow0MUZng=', 'Content-Length': '0'} 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd4144fd1b2f7fb0-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.470 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj36 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019d4ce635d20ef68-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.473 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36?versionId=null 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc= 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4og8uNECK+Xi7Dsb8+Ek4a8S5Mc=', 'Content-Length': '0'} 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c80af881f059b942-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d616a5cb7b8c08-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000051b111d53951044-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.478 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA= 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA=', 'Content-Length': '0'} 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055b0ee13cefa6c46-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA= 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T3UN0Cnq0OcJ00yLce/7VuftyxA=', 'Content-Length': '0'} 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004009840f6155219d-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj36 2024-04-10T00:01:37.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.481 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM= 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM=', 'Content-Length': '0'} 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045d9e1e52f94c970-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj36 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj36 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj36 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj36 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM= 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hsH1hE55S6cpgAij2QAoGlLqqYM=', 'Content-Length': '0'} 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094ed55767f58bc0e-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.484 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff248fb2f61dfa96-006615b8aa-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj36 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw= 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zS6IltS9iQe85xSDLCvKDKadbWw=', 'Content-Length': '0'} 2024-04-10T00:01:37.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c614a57dd3515b-006615b8aa-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj361000falseobj362024-04-09T21:48:02.600Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI= 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI=', 'Content-Length': '0'} 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000402c8d30d921e9f0-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.489 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI= 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6Py3eEPX7OE4UqLD8sUIeMxZ/zI=', 'Content-Length': '0'} 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009679fca4eb8ddc96-006615b8aa-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:42 GMT 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Y7X1tfe811ftdYY8Eyyb5GThhjM= 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y7X1tfe811ftdYY8Eyyb5GThhjM=', 'Content-Length': '0'} 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fc56e47d68273b4-006615b8aa-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:aZyfdRdPmHP19bXLpGDiCSh+Fzg= 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aZyfdRdPmHP19bXLpGDiCSh+Fzg=', 'Content-Length': '0'} 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6478ba9ed29b903-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-10T00:01:37.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.494 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8= 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8=', 'Content-Length': '0'} 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cc4f272f9014a2d-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37?versionId=null 2024-04-10T00:01:37.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8= 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2gYqcn/RK5RFuPbmxmHP8Sdz1s8=', 'Content-Length': '0'} 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040b4353d6b56e398-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092285209e1a5365a-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.499 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270ef9dc7430ba99-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.501 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0= 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0=', 'Content-Length': '0'} 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce0fff2fc0387615-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0= 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:g/TkZQJ5LOJi6s0EhuzBNMhWVl0=', 'Content-Length': '0'} 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000745ce149aa237b58-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj37 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.504 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o= 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o=', 'Content-Length': '0'} 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a84c1dab9938791-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj37 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj37 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj37 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj37 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o= 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nKRgUiXtmOlnvmXiMfsudjv2m0o=', 'Content-Length': '0'} 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e3eb6a7f59a7de3-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.507 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6d807bec66b70e4-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj37 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1bb8b19b889103-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj371000falseobj372024-04-09T21:48:02.648Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e80653824eaa3c6-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.512 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b86f6d5c554a2cf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws= 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws=', 'Content-Length': '0'} 2024-04-10T00:01:37.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000109190efc73f0805-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws= 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t/mqSFdeOZVeam4dq7yFuGWV2ws=', 'Content-Length': '0'} 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b69ed1678f1346ed-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.517 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097c2e63c9a29ea32-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38?versionId=null 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58= 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7CtRDNUX9H+mYurHUBC2tYTDm58=', 'Content-Length': '0'} 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b01dadcc9886a473-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.520 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000819f3db4749d4287-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000911baa059a025679-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.523 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc= 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc=', 'Content-Length': '0'} 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f38e79c84fa6730-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc= 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/oX2LUKuiagadrW6QrSMLR6n3Tc=', 'Content-Length': '0'} 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bd96acc4e81f004-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj38 2024-04-10T00:01:37.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.526 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g= 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g=', 'Content-Length': '0'} 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d210b15343a48c7-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj38 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj38 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj38 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj38 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g= 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TgKU6AgQ38N2s71HYFwHpJvON8g=', 'Content-Length': '0'} 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055aa96410e341041-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.529 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090a40c63c1c520d0-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj38 2024-04-10T00:01:37.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e90b496223dfa7a-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj381000falseobj382024-04-09T21:48:02.657Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj39 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013ce626d5606d21e-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.534 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076f2eaa2b04afe5f-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU= 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU=', 'Content-Length': '0'} 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002865f6a3daafa27e-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU= 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yjiUsvBHnoRqvzF2J5ZdoWWaGzU=', 'Content-Length': '0'} 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b319e0fa839dfccf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj39 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.539 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cd996858eee2f1b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39?versionId=null 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg= 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:h2ltwchxEfwWuMojAQyKSd25Usg=', 'Content-Length': '0'} 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013d9919892657597-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2ddd0fd161c94a-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.544 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d09c93176c8af1c3-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA= 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA=', 'Content-Length': '0'} 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001250bce39bff58b2-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA= 2024-04-10T00:01:37.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MuuFNlEMJg6993sJcXNwf2tZMCA=', 'Content-Length': '0'} 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa21be20f20e943d-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj39 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.548 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ= 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076371154d9cb4312-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj39 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj39 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj39 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj39 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ= 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IPTA7xs9Y+79pogWUDV0zZk0GgQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c61f90e4ec486ce8-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.551 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1288f2a5736fd4-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj39 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015ff77c296217def-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj391000falseobj392024-04-09T21:48:02.704Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj4 2024-04-10T00:01:37.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.554 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005067dd8464622edf-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-10T00:01:37.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e275bfc3b73e19a-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.557 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I= 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I=', 'Content-Length': '0'} 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2dfa7f8279d3a5a-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I= 2024-04-10T00:01:37.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:66sBL8ZIie6d2VX9MUjxirHHM8I=', 'Content-Length': '0'} 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038e8b0fa85d41594-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj4 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a280f83cf851ac82-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4?versionId=null 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.562 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8= 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bsbfJGQyf+7+YpWmAlYYuNWDX+8=', 'Content-Length': '0'} 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fb948901bb15ddf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001215a650ce5dda9d-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.565 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce32dd4921f3b618-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE= 2024-04-10T00:01:37.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE=', 'Content-Length': '0'} 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000531297b9870523ed-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.568 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE= 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Kq+IZ/UuX6+8NNVVn6POmYhehIE=', 'Content-Length': '0'} 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48a86fd3321d638-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj4 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4 2024-04-10T00:01:37.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.570 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA= 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA=', 'Content-Length': '0'} 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b5b4ff62288c77-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj4 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj4 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj4 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj4 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA= 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JLvGoRgi1FMAL5B2xDU2JUf8IPA=', 'Content-Length': '0'} 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055b1029cd98276e0-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.573 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1d63faaea8559f9-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj4 2024-04-10T00:01:37.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007310ae2a4bf3d60e-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj41000falseobj42024-04-09T21:48:01.794Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj40 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005acc8c4ceaed58fd-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.578 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b6d7d68b4d4c233-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8= 2024-04-10T00:01:37.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8=', 'Content-Length': '0'} 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002196d2a2e362b038-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.581 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8= 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AG7htuJyWDso17L1wyL1s9VcSe8=', 'Content-Length': '0'} 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000494c5e6144603cd1-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj40 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.583 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a59898f9f466dd50-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40?versionId=null 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY= 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OJT4Mc6uvCmY5AIPOkE2Ny+6XlY=', 'Content-Length': '0'} 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eb56693d3ef1c95-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.586 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b32819dd4f4edc-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079361c2e3a11b584-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w= 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w=', 'Content-Length': '0'} 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f4b0ceb134eb0bc-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.591 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w= 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y9Ia52iRBmXgea3eMpyi0UZH55w=', 'Content-Length': '0'} 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000577607b3ae10b657-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj40 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI= 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI=', 'Content-Length': '0'} 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b722cb1dbe26c71-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj40 2024-04-10T00:01:37.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj40 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj40 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.594 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj40 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI= 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gtkNqX088C79/6Dw9zd5AStUGHI=', 'Content-Length': '0'} 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0c1188aa7635e26-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5226e971a8e9758-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj40 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceaa7f37fc65d2a9-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj401000falseobj402024-04-09T21:48:02.712Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj41 2024-04-10T00:01:37.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.599 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ffa76a1691c583a-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d65a3919a14ffa3a-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.602 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU= 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU=', 'Content-Length': '0'} 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbcb8c25f191c965-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU= 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FpL28Jd5hjDpybLXxxnvHHeDblU=', 'Content-Length': '0'} 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050a29af3044c7c8c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj41 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b309951a28142c59-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.607 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41?versionId=null 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA= 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0vavokyeuu5t20Qgc5WT48Ox+bA=', 'Content-Length': '0'} 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086b920ed2d5f213d-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab28008771d83a6b-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067132e261c2a7af4-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA= 2024-04-10T00:01:37.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA=', 'Content-Length': '0'} 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000d1769c6c3e5f94-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.613 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA= 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u4L/XXd01NUpCCHf2tvMyPPm4BA=', 'Content-Length': '0'} 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000babc05c1b14ebccf-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj41 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41 2024-04-10T00:01:37.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI= 2024-04-10T00:01:37.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI=', 'Content-Length': '0'} 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dacf7ca5ecc66aea-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj41 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj41 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj41 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.616 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj41 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI= 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7uCwiyL66EBrxIoCqryeH5HhuMI=', 'Content-Length': '0'} 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093e605ac75674c9c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da059099dec481e-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj41 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000787486faf2c3395a-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj411000falseobj412024-04-09T21:48:02.720Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj42 2024-04-10T00:01:37.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.621 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce5886c5f291ab61-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008aef2171073ed7b7-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q= 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q=', 'Content-Length': '0'} 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000064d04266b62f54c7-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.626 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q= 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7MRLzCBL/YrWYjOH9rXqqbsor8Q=', 'Content-Length': '0'} 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000814f4be9f4c6be2c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj42 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000197e601189b1155b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42?versionId=null 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA= 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UW+Y98gvbqQaoIbRiOf28XtvVJA=', 'Content-Length': '0'} 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b22000df5db56301-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.631 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dab7d5294b57c24-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0aee3b7f1187224-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk= 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk=', 'Content-Length': '0'} 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045d1d5418ec977a5-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42 2024-04-10T00:01:37.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk= 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:npgRJX21zQ3dbMNSHcuLl/FqZLk=', 'Content-Length': '0'} 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fe9f30210faf905-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj42 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM= 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM=', 'Content-Length': '0'} 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000751c6cc5e5e2839a-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj42 2024-04-10T00:01:37.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj42 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj42 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.639 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj42 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM= 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qM8mS+1TPotqb5RquFlqooJ54GM=', 'Content-Length': '0'} 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aef050f28ae63e3a-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa2b0f125958c11-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj42 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.642 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bcb3ce488eaf2e5-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj421000falseobj422024-04-09T21:48:02.769Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj43 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.644 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c1d6ff890d0e298-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-10T00:01:37.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000882f776e27d82707-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI= 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI=', 'Content-Length': '0'} 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009522b2c4dd84e076-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI= 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XDBwx5X5fIQ8hdnyGJZDs4qPJCI=', 'Content-Length': '0'} 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097fe60549ad57d5f-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj43 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-10T00:01:37.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ce699e15b673efa-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.652 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43?versionId=null 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE= 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cLhxPWOKe4e8dlxFM744z3J00BE=', 'Content-Length': '0'} 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7a91b3f1fd6ed61-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.654 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0e9902488490a4b-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040d683728346dfb3-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.657 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43 2024-04-10T00:01:37.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.658 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw= 2024-04-10T00:01:37.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw=', 'Content-Length': '0'} 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fb69a31fbb07a38-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw= 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:KDpWs+xSGjw5MBu4FGeZodzy1pw=', 'Content-Length': '0'} 2024-04-10T00:01:37.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000252835fa513f31ac-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj43 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.663 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE= 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE=', 'Content-Length': '0'} 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073992b58cb76d18b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj43 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj43 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj43 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj43 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE= 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mB8sv4La86Rvaq0qW1111jfjmyE=', 'Content-Length': '0'} 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e901e474d2776850-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.666 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000700ff1e60d6659ac-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj43 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.668 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7b594c725a233dc-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj431000falseobj432024-04-09T21:48:02.777Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-10T00:01:37.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9f072a70a30094f-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048fd593cbde71273-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.673 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc= 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc=', 'Content-Length': '0'} 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005bd007a2fe0c02e3-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.675 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc= 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IVAespD6k13r2AGC1c8Hwzg3znc=', 'Content-Length': '0'} 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8940b62dca26242-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-10T00:01:37.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000078c10c7fb134e11b-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44?versionId=null 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8= 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:v+8mkcgcRBPzVNUTFlMqHbvyDn8=', 'Content-Length': '0'} 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007717b4418e1961d5-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.680 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adb10268c9692cb4-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.682 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f85962ff8796273-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas= 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas=', 'Content-Length': '0'} 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007e40557f6a5dd2e-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44 2024-04-10T00:01:37.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas= 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GU2gjd3zEZIpSwBjmnPO0A9yLas=', 'Content-Length': '0'} 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c644c91e82d01a2d-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc= 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc=', 'Content-Length': '0'} 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7ad537409d1e93f-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj44 2024-04-10T00:01:37.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj44 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj44 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.688 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj44 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc= 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wKmKd/KWlVum/vpMDjva8WS5dIc=', 'Content-Length': '0'} 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb737347e3608bfd-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eeeb7e2d11a74736-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj44 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdf7cddcb7fbe5a-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj441000falseobj442024-04-09T21:48:02.824Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj45 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.693 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000740e8f9affb39317-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.695 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c50b02a1978ac1c9-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ= 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001db6e5e1ae817975-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.698 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ= 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:64ATgAINgQ0VdiZl366IAWECLTQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ecb4d188f398c78-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj45 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b76a68adf856159f-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45?versionId=null 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE= 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b8o58nfXmWzdIdBaeEZhRpZz3CE=', 'Content-Length': '0'} 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fafd43da2718343-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.703 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8c7ec34b7ac310c-006615b8ab-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE= 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uqJ4p8TckkXZx0vy/N9V+wzYadE=', 'Content-Length': '0'} 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007317b61567096103-006615b8ab-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI= 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI=', 'Content-Length': '0'} 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040bf1fe58492f1d9-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.708 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI= 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DGqe9DCCV0+ZQggJnL4r2PWfNUI=', 'Content-Length': '0'} 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d73d26fc82a843c-006615b8ab-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj45 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:43 GMT 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:leQpFXMkn9G639FjF3AMs13u2Ko= 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:leQpFXMkn9G639FjF3AMs13u2Ko=', 'Content-Length': '0'} 2024-04-10T00:01:37.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d3e26b3a1c551697-006615b8ab-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj45 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj45 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj45 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.711 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj45 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xKxiIkPHA9ioIiNE285KdoHqELQ= 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xKxiIkPHA9ioIiNE285KdoHqELQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a473a7aa383d2b03-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.713 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080fea8c619fc6b17-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj45 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007c4f2c08726f6c3-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj451000falseobj452024-04-09T21:48:02.833Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb6cf9fc116f97fe-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.718 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057550b00c3381177-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.720 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo= 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo=', 'Content-Length': '0'} 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d0d07e210cd8e6c-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo= 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nVwrz/EFEaLvf4R4kl8PmHIDNQo=', 'Content-Length': '0'} 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021cea141023f85c3-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.722 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008149bb0e1a0de1a-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.725 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46?versionId=null 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE= 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UCiBcuQ85gP6EW0T4N+zw2v70FE=', 'Content-Length': '0'} 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011ed7d5bb5cc62fc-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004263b1263478f1ad-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.728 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dca8e84c37ba1a7-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.730 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o= 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o=', 'Content-Length': '0'} 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f46e1732fdb59a99-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o= 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WL8Qb8XDxcH/qfKnZeHBquvLH2o=', 'Content-Length': '0'} 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c6c55b5672e6ed5-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj46 2024-04-10T00:01:37.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.733 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA= 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA=', 'Content-Length': '0'} 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087cd5c16577e3732-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj46 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj46 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj46 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj46 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA= 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HG/cGVy3flQGB6rvGSV4AgMtRiA=', 'Content-Length': '0'} 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e675822cbf9824e3-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.736 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049c567bad725bdac-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj46 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.738 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa7edf0634cf45ce-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj461000falseobj462024-04-09T21:48:02.881Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj47 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.740 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035349c4d4d3190c6-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d08a461689d92c66-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.743 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY= 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY=', 'Content-Length': '0'} 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069bba0a693735853-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY= 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqY9jrXkLUgD2lHz61xPn5OOXWY=', 'Content-Length': '0'} 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000722811ce80b38e87-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj47 2024-04-10T00:01:37.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fa322ae9fea02e0-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.748 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47?versionId=null 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww= 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1XHhb+X4NKjjzqMtyrLIEGDP2ww=', 'Content-Length': '0'} 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073d8ed48fe22b12c-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd968ae5f69b3bda-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ada15991c199121-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.753 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM= 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM=', 'Content-Length': '0'} 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cf05b727c46b7fd-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM= 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i415Rf2Sch6b8GFER9cxXu8rgTM=', 'Content-Length': '0'} 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba00d43330d67d5e-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj47 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47 2024-04-10T00:01:37.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.756 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M= 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M=', 'Content-Length': '0'} 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5b476ec3f08e1f8-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj47 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj47 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj47 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.757 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj47 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M= 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fNYMdqN+t6HO2f+BsFIoIvtxG7M=', 'Content-Length': '0'} 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000031778d4679dae19-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000da4630c1034151c-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj47 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e556658300ef1e3e-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj471000falseobj472024-04-09T21:48:02.889Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj48 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000441f36e72a1f603e-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.764 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002faa72a8fff2540a-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.766 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg= 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg=', 'Content-Length': '0'} 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000488181543a9eaf3f-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg= 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YouDX1lmvAbHXNhlieDOofYPuUg=', 'Content-Length': '0'} 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bac3d2f231b6974-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj48 2024-04-10T00:01:37.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.769 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086e195f66cf4d2a6-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48?versionId=null 2024-04-10T00:01:37.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY= 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Fxw6RykhsJU22d1w4hiGaO/GEcY=', 'Content-Length': '0'} 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a019a37d663a5b3-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008953b47e0790fed3-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.774 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b5e7944215e2d8f-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo= 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo=', 'Content-Length': '0'} 2024-04-10T00:01:37.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eecd24d28128236e-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.777 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo= 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r2Z1NvV+o6KS/vk2O8TNgexXBMo=', 'Content-Length': '0'} 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b53542bf228fcf45-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj48 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48 2024-04-10T00:01:37.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw= 2024-04-10T00:01:37.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw=', 'Content-Length': '0'} 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e099ce3720ff796a-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj48 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj48 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj48 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.780 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj48 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw= 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cDcP2eRn4O73ZH0LCYXeQv623Xw=', 'Content-Length': '0'} 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6eef1a686bcc60c-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.782 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676da375805812cc-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj48 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb229b6eed8e1bf0-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj481000falseobj482024-04-09T21:48:02.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj49 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000955d05aa46cc4789-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.787 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031eb36bacb00398e-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U= 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U=', 'Content-Length': '0'} 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000028af6574839e9e34-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.790 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U= 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+PbYDy2oMdbskecSC4s3yhm1/2U=', 'Content-Length': '0'} 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001af6ca1814c151e4-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj49 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.792 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707d47179f994eeb-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49?versionId=null 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k= 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nbV4KqPh675mfgVoDu7pn24xE7k=', 'Content-Length': '0'} 2024-04-10T00:01:37.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc15cc6425ec406-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.795 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000087ac07b43fadee6-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.797 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005919b22267968776-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw= 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw=', 'Content-Length': '0'} 2024-04-10T00:01:37.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000095ffbaa53870867-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.800 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw= 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2FWsZY+LArnVm92ODXUXtLGpjbw=', 'Content-Length': '0'} 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a06d0207a78f1e69-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj49 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c= 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c=', 'Content-Length': '0'} 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8fe037875367546-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj49 2024-04-10T00:01:37.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj49 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj49 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj49 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.803 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c= 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UcygzJ0KkToKvYw3oknAx5LNA1c=', 'Content-Length': '0'} 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f6d20930d385e07-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0556c030efa7a1c-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj49 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.806 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076fae432474d9062-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj491000falseobj492024-04-09T21:48:02.944Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj5 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047e9228716106061-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.809 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000451e63f707f197e4-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ= 2024-04-10T00:01:37.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000087e0cec750d0c39-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ= 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:M+580QapTSawVDb4WeBQSbpLpfQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009662b7c87c971f41-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj5 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.814 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a30b1dcbd17873c0-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5?versionId=null 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k= 2024-04-10T00:01:37.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WWySenMnszFkEcbNoDSCJg4888k=', 'Content-Length': '0'} 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fbf782e47da43cf-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162254f3f765c2e2-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.819 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a468290ed320ff84-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg= 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg=', 'Content-Length': '0'} 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbf0f44b20ac0e4b-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.822 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg= 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kVJUCACBHyTLVjAEu5CNkEwFcPg=', 'Content-Length': '0'} 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000049ddfbf9f21f23d-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj5 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo= 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo=', 'Content-Length': '0'} 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b72f7b573fda36e2-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj5 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj5 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj5 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.825 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj5 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo= 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:znqm0Dw3Jnvrc/kEZ8yM5lwElVo=', 'Content-Length': '0'} 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ad355b9e5f7601e-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2e84cd9967c4812-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj5 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.828 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c96c22998ef9ebf7-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj51000falseobj52024-04-09T21:48:01.840Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj6 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003477893755c5c9a8-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a194e85ce7f5332a-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.833 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo= 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo=', 'Content-Length': '0'} 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c75ca16a87638544-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.835 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo= 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lITgnj8ohtp9qr1x86IEOF2sDSo=', 'Content-Length': '0'} 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb2e51f6cf57e396-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj6 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7e2838190c91855-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.838 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6?versionId=null 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg= 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SdKSpr4h7MP+PsgP52ykO1a9uwg=', 'Content-Length': '0'} 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097acaac7334fdb4a-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8c3c76a17e37a9-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3a9db85baac754-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.843 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8= 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8=', 'Content-Length': '0'} 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dda5b03fdb20d7c-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8= 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EF4nAmTAJt9AYg+GC/ty6gau3u8=', 'Content-Length': '0'} 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adce83c7eb242f48-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.845 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj6 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.846 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4= 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4=', 'Content-Length': '0'} 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c3cc076e00e3eaa-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj6 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj6 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj6 2024-04-10T00:01:37.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj6 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4= 2024-04-10T00:01:37.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ja0B8WYYMTaP+lyAYw1NjkKLMQ4=', 'Content-Length': '0'} 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cfe7d681bbcbabad-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008304f812ea637ec7-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj6 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.851 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d136c58100e98160-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj61000falseobj62024-04-09T21:48:01.849Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj7 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-10T00:01:37.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014c1dad13302cc00-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075e52f1adb5c2ffa-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8= 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8=', 'Content-Length': '0'} 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018bdff3660850448-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8= 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:68x/ulnXWuzt+6JTVrIgeSwhTO8=', 'Content-Length': '0'} 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077467ffe8de1fda7-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.858 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj7 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.859 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e7f79e82c3ba80d-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7?versionId=null 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.861 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM= 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:X7JiJEYe0VJqhy0A4VBLyoy0SJM=', 'Content-Length': '0'} 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb9e734ab0d036da-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312f6c78fc37fbb9-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.864 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359d250969614b89-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ= 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a1807f717067148-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.867 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ= 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:w8Pc+QSGz1rehb9k/UOHk4MBUjQ=', 'Content-Length': '0'} 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c3884f2145feaca-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj7 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7 2024-04-10T00:01:37.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY= 2024-04-10T00:01:37.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY=', 'Content-Length': '0'} 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab0075c2aa674e50-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj7 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj7 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj7 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.870 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj7 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY= 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+w8HKc875VFSnaTzi4iwYznSnRY=', 'Content-Length': '0'} 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000067674d77c085512-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.872 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007714382d4499480a-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj7 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003648e11dfc4427a6-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj71000falseobj72024-04-09T21:48:01.901Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj8 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095096bb2e6d81f28-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.877 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000061f6376d4f8abae-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA= 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA=', 'Content-Length': '0'} 2024-04-10T00:01:37.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cea0c98201b137c-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.880 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA= 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+9I9NeEcQCVq3nuoqo3k4sXhzoA=', 'Content-Length': '0'} 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b15279e676555cf-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj8 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d972f6672f559ff4-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8?versionId=null 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI= 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9MEVXc4710pIkrDueHC7hdlcwgI=', 'Content-Length': '0'} 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016663f98cc1a05da-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.885 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3118500ba23c0c0-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000960b6a9a36dc459c-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g= 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g=', 'Content-Length': '0'} 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000867fecc7d726ddab-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.890 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g= 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7Ph5VC4S3LSAIYfNZoPY0D1Rq9g=', 'Content-Length': '0'} 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb366880f53fbd8-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj8 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8= 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8=', 'Content-Length': '0'} 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b1757000e445a681-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj8 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj8 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj8 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.893 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj8 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8= 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jz8lZb8b66fkuhaiY0s+8luHZf8=', 'Content-Length': '0'} 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000492f99d3037ac1a8-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb51a34cec2c6896-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj8 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a2f6b1e5585eed3-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj81000falseobj82024-04-09T21:48:01.909Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj9 2024-04-10T00:01:37.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.898 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a34efa8a1f0bbe97-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b56d1eb148427b18-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.901 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM= 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM=', 'Content-Length': '0'} 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091f2eac27e5bc4f8-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.903 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM= 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:eN/XbV/MZvzVUvXR3vqaXYrVwfM=', 'Content-Length': '0'} 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000061085883d1b68c48-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj9 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e685693f676fa972-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.906 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9?versionId=null 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE= 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mq54V6CYJ4q18XGno6jff5RcQjE=', 'Content-Length': '0'} 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000326fd855d00941e4-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.910 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b46303b64f883c79-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063e2685bbadaaba5-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.913 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc= 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc=', 'Content-Length': '0'} 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000828c5ecc11181c6b-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc= 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZC6hHEuQEoxhyFn+ReVndFikhBc=', 'Content-Length': '0'} 2024-04-10T00:01:37.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062ad1c9c1bafc68d-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj9 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.916 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI= 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI=', 'Content-Length': '0'} 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-11:a5723776-f142-428c-a6df-d7bf949257da.4372.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f71c034dc3cdae27-006615b8ac-1112-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/obj9 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/obj9 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/obj9 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:/xryqqk-11/obj9 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.918 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI= 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:FzFibMElPEVC5ODruicoWVFVAqI=', 'Content-Length': '0'} 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 21:48:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d1af103f3207fb9-006615b8ac-1110-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000113249aa343b9c6d-006615b8ac-1112-a2'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/ 2024-04-10T00:01:37.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/ 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-11/?prefix=obj9 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:37.921 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 21:52:44 GMT 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:/xryqqk-11/ 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc= 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 21:52:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r8tvScZFav2YOt0rnSBEIO0bwoc=', 'Content-Length': '0'} 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c13ec758fc2b816f-006615b8ac-1110-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 21:52:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-11obj91000falseobj92024-04-09T21:48:01.957Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=xryqqk-11 zones={a2, a1} 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:37.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:44.981+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:49.981+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:54.982+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:52:59.983+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:04.983+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:09.984+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:14.985+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:19.986+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:24.987+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:29.986+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:34.987+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:39.987+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:44.988+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:49.989+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:54.989+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:53:59.990+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:04.991+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:09.991+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:14.992+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:19.993+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:24.993+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:29.994+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:34.994+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:39.995+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:44.995+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:49.997+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:54.997+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:54:59.998+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:04.999+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:09.998+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:14.999+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:20.000+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:25.000+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:30.001+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:35.001+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:40.002+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:45.003+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:50.003+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:55:55.004+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:00.005+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:05.005+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:10.006+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:15.006+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:20.007+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:25.008+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:30.008+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:35.010+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:40.010+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:45.010+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:50.011+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:56:55.011+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:00.012+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:05.012+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:10.013+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:15.014+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:20.014+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:25.015+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:30.016+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:35.016+0000 7f8bdefdb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:40.017+0000 7f8bdefdb780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:40.017+0000 7f8bdefdb780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T21:57:40.017+0000 7f8bdefdb780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:37.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:37.928 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:37.928 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.929 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:37.929 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-04-10T00:01:37.929 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-10T00:01:37.930 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-33 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-33/ 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-33/ 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-33/ 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:07:08 GMT 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:/xryqqk-33/ 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:t0Dyv1juq9uPR1bpCu0iIykDnpE= 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:t0Dyv1juq9uPR1bpCu0iIykDnpE=', 'Content-Length': '0'} 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000001cc64436c6383c10-006615bc0c-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-34 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-34/ 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-34/ 2024-04-10T00:01:37.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-34/ 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:07:08 GMT 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:/xryqqk-34/ 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:37.933 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ljVltV4r3La8cU0Ve4AQDctjmck= 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:07:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ljVltV4r3La8cU0Ve4AQDctjmck=', 'Content-Length': '0'} 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000074b0f3b647de9631-006615bc0c-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:07:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700428.214372_408.1", 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:07:08.214372Z" 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.935 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:08.459+0000 7f34d1156780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:08.460+0000 7f34d1156780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:37.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:37.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.586841_338.1", 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.586841Z", 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:08Z" 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:37.941 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.943 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.586841_338.1", 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.586841Z", 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.944 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.945 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:08Z" 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.586841_338.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.586841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.586841_338.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.586841_338.1 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:13.635+0000 7f5922fd0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:13.637+0000 7f5922fd0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:37.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.555171_334.1", 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:13Z" 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:37.954 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.555171_334.1", 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:13Z" 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.555171_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.555171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:37.957 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.555171_334.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.555171_334.1 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:18.736+0000 7f649554f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:18.737+0000 7f649554f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:37.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.539236_330.1", 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:37.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:37.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:18Z" 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:37.963 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:37.964 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.965 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.539236_330.1", 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.967 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:18Z" 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.539236_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.539236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.539236_330.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.539236_330.1 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:23.844+0000 7fc122569780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:23.845+0000 7fc122569780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:23Z" 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:37.973 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.974 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.975 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:37.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.977 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:23Z" 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:28.962+0000 7f80ca4fd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:28.973+0000 7f80ca4fd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:37.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.539236_330.1", 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:28Z" 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:37.984 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.986 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.539236_330.1", 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.987 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.988 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:28Z" 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.539236_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.539236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.539236_330.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.539236_330.1 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.989 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:34.103+0000 7fc0f3cd5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:34.104+0000 7fc0f3cd5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.542449_332.1", 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.542449Z", 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:37.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:37.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:34Z" 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:37.995 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.996 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:37.997 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.542449_332.1", 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.542449Z", 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:37.998 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:37.999 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:34Z" 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.542449_332.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.542449Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.542449_332.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.542449_332.1 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:39.205+0000 7fe3e9fbc780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:39.206+0000 7fe3e9fbc780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.622748_342.1", 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.622748Z", 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:39Z" 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.005 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.006 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.007 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.622748_342.1", 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.622748Z", 2024-04-10T00:01:38.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.009 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:39Z" 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.622748_342.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.622748Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.010 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.622748_342.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.622748_342.1 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:44.308+0000 7f69e53c6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:44.309+0000 7f69e53c6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.584285_337.1", 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:44Z" 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.016 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.017 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.018 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.584285_337.1", 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.019 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.020 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:44Z" 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.584285_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.584285Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.584285_337.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.584285_337.1 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:49.455+0000 7f77f50c6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:49.456+0000 7f77f50c6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.584285_337.1", 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:49Z" 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.026 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.027 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.028 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.584285_337.1", 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.030 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:49Z" 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.584285_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.584285Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.584285_337.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.584285_337.1 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:54.659+0000 7fada877d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:54.661+0000 7fada877d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.555171_334.1", 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:54Z" 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.037 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.038 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.039 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.555171_334.1", 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.040 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:54Z" 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.041 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.555171_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.555171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.555171_334.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.555171_334.1 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:59.768+0000 7fe4fcbcd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:07:59.769+0000 7fe4fcbcd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.625978_344.1", 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.625978Z", 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:07:59Z" 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.047 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.048 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.049 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.625978_344.1", 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.625978Z", 2024-04-10T00:01:38.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.051 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:07:59Z" 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.625978_344.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.625978Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.625978_344.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.625978_344.1 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:04.869+0000 7fd49a131780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:04.870+0000 7fd49a131780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.639793_345.1", 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.639793Z", 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:04Z" 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.058 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.059 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.060 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.639793_345.1", 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.639793Z", 2024-04-10T00:01:38.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.062 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:04Z" 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.639793_345.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.639793Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.639793_345.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.639793_345.1 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:09.984+0000 7f22ec8e4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:09.985+0000 7f22ec8e4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.064 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.065 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.555171_334.1", 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-10T00:01:38.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:09Z" 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.069 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.070 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.071 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.555171_334.1", 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.072 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.555171Z", 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.073 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.074 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:09Z" 2024-04-10T00:01:38.075 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.075 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.555171_334.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.555171Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.555171_334.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.555171_334.1 2024-04-10T00:01:38.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.076 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.076 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:15.190+0000 7f0009262780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:15.194+0000 7f0009262780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.558529_336.1", 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.558529Z", 2024-04-10T00:01:38.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:15Z" 2024-04-10T00:01:38.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.089 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.089 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.089 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.089 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.089 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.090 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.091 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.558529_336.1", 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.558529Z", 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:15Z" 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.558529_336.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.558529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.558529_336.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.558529_336.1 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:20.299+0000 7f1293bb9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:20.300+0000 7f1293bb9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:20Z" 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.101 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.102 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.105 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.107 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.108 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:20Z" 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.675822_352.1 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:25.403+0000 7f0e0c9ec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:25.404+0000 7f0e0c9ec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.540137_331.1", 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.540137Z", 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:25Z" 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.116 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.117 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.118 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.540137_331.1", 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.540137Z", 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.119 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.120 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:25Z" 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.540137_331.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.540137Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.540137_331.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.540137_331.1 2024-04-10T00:01:38.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:30.505+0000 7f339b945780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:30.506+0000 7f339b945780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.558529_336.1", 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.558529Z", 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:30Z" 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.127 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.128 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.558529_336.1", 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.558529Z", 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.130 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:30Z" 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.558529_336.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.558529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.558529_336.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.558529_336.1 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:35.619+0000 7fab881ef780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:35.620+0000 7fab881ef780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.539236_330.1", 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:35Z" 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.137 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.139 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.539236_330.1", 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.539236Z", 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.140 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.141 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:35Z" 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.539236_330.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.539236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.539236_330.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.539236_330.1 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:40.732+0000 7fdaafae3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:40.733+0000 7fdaafae3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.620202_341.1", 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.620202Z", 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:40Z" 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.147 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.148 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.620202_341.1", 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.620202Z", 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:40Z" 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.620202_341.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.620202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.620202_341.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.620202_341.1 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:45.932+0000 7fe7128c3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:45.934+0000 7fe7128c3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.584285_337.1", 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:45Z" 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.159 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.584285_337.1", 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.584285Z", 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.161 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:45Z" 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.584285_337.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.584285Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.584285_337.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.584285_337.1 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:51.050+0000 7f687c824780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:51.051+0000 7f687c824780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.556117_335.1", 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.556117Z", 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:51Z" 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.168 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.169 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.556117_335.1", 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.556117Z", 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:51Z" 2024-04-10T00:01:38.172 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.556117_335.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.556117Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.556117_335.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.556117_335.1 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:56.153+0000 7fd91e0fd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:08:56.154+0000 7fd91e0fd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:08:56Z" 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.178 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.180 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.181 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:08:56Z" 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:01.266+0000 7f0257e34780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:01.267+0000 7f0257e34780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:01Z" 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.189 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:01Z" 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.193 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:06.481+0000 7fd197416780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:06.482+0000 7fd197416780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:06Z" 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.199 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.201 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.203 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:06Z" 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.204 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:11.589+0000 7fb311f41780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:11.590+0000 7fb311f41780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:11Z" 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.210 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.211 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:11Z" 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.214 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:16.709+0000 7fded642a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:16.711+0000 7fded642a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:16Z" 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.220 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.221 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:16Z" 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:21.830+0000 7f7fc112f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:21.834+0000 7f7fc112f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:21Z" 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.230 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.232 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.233 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.234 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:21Z" 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:26.996+0000 7fce4f155780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:26.997+0000 7fce4f155780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:26Z" 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.241 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.242 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.243 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.244 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:26Z" 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:32.105+0000 7f4fd3f68780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:32.107+0000 7f4fd3f68780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:32Z" 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.251 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.252 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.255 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:32Z" 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:37.216+0000 7f41f3b41780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:37.217+0000 7f41f3b41780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:37Z" 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.261 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.262 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.264 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.265 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:37Z" 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.266 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.874323_373.1 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:42.324+0000 7fd52f51b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:42.326+0000 7fd52f51b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.691210_354.1", 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.691210Z", 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:42Z" 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.273 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.274 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.691210_354.1", 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.691210Z", 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.275 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.276 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:42Z" 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.691210_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.691210Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.691210_354.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.691210_354.1 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:47.434+0000 7ffa3da22780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:47.435+0000 7ffa3da22780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.691210_354.1", 2024-04-10T00:01:38.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.691210Z", 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:47Z" 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.282 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.283 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.284 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.691210_354.1", 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.691210Z", 2024-04-10T00:01:38.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.286 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:47Z" 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.691210_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.691210Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.691210_354.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.691210_354.1 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:52.547+0000 7f8874792780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:52.548+0000 7f8874792780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:52Z" 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.293 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.294 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.295 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.296 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.297 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:52Z" 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.692156_355.1 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:57.684+0000 7fdbab0ec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:09:57.685+0000 7fdbab0ec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:09:57Z" 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.303 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.304 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.305 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.306 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:09:57Z" 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:02.788+0000 7f1502bac780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:02.789+0000 7f1502bac780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:02Z" 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.314 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:02Z" 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:07.911+0000 7ff748698780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:07.912+0000 7ff748698780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:07Z" 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.325 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.326 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.328 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:07Z" 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.329 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:13.015+0000 7fef222c2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:13.016+0000 7fef222c2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:13Z" 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.335 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.336 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:38.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:13Z" 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.675822_352.1 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:18.132+0000 7f3441762780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:18.133+0000 7f3441762780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.707433_359.1", 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.707433Z", 2024-04-10T00:01:38.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:18Z" 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.345 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.346 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.707433_359.1", 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.707433Z", 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.349 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:18Z" 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.707433_359.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.707433Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.707433_359.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.707433_359.1 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:23.246+0000 7f4bcc07e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:23.247+0000 7f4bcc07e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:23Z" 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.356 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:23Z" 2024-04-10T00:01:38.360 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:28.418+0000 7f94866dd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:28.419+0000 7f94866dd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:28Z" 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.366 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.367 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.370 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:28Z" 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:33.526+0000 7f3269c09780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:33.527+0000 7f3269c09780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:33Z" 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.379 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.381 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:33Z" 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:38.632+0000 7f556fb81780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:38.632+0000 7f556fb81780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:38Z" 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.387 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.390 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:38Z" 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:43.737+0000 7f9874a04780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:43.739+0000 7f9874a04780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:43Z" 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.398 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.399 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:43Z" 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:48.845+0000 7fbbc15e8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:48.846+0000 7fbbc15e8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:48Z" 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.408 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.410 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:48Z" 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.673463_351.1 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:53.953+0000 7f6ba274c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:53.954+0000 7f6ba274c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:53Z" 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.418 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.419 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:53Z" 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.422 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:59.146+0000 7f55d25ea780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:10:59.147+0000 7f55d25ea780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:10:59Z" 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.428 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.429 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.430 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.431 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.432 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.433 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:10:59Z" 2024-04-10T00:01:38.433 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.433 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.433 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:04.250+0000 7f0284b9b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:04.251+0000 7f0284b9b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:04Z" 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.440 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.441 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.444 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:04Z" 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:09.353+0000 7f4095eab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:09.354+0000 7f4095eab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:09Z" 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.451 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:09Z" 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:14.470+0000 7fef2f1b4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:14.471+0000 7fef2f1b4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:14Z" 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.463 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.465 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:14Z" 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.673463_351.1 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:19.586+0000 7f03c0dbc780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:19.587+0000 7f03c0dbc780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:19Z" 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.473 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.475 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:19Z" 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.692156_355.1 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:24.691+0000 7f51ea741780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:24.692+0000 7f51ea741780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:24Z" 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.482 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.483 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.486 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:24Z" 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:29.793+0000 7f3610144780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:29.794+0000 7f3610144780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:29Z" 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.492 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:29Z" 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:38.497 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:34.906+0000 7f547bac8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:34.907+0000 7f547bac8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:34Z" 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.504 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.506 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:34Z" 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:40.022+0000 7f909ebb2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:40.023+0000 7f909ebb2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:40Z" 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.514 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.515 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.516 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.517 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:40Z" 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:45.193+0000 7f8c4e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:45.194+0000 7f8c4e6ac780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:45Z" 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.525 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:45Z" 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.730558_363.1 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:50.295+0000 7f36a6e0d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:50.296+0000 7f36a6e0d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:50Z" 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.537 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:50Z" 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.874323_373.1 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:55.409+0000 7f4952ad8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:11:55.410+0000 7f4952ad8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:11:55Z" 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.544 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.545 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.548 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:11:55Z" 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:00.520+0000 7f1fc6383780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:00.521+0000 7f1fc6383780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:00Z" 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.555 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.557 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:00Z" 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.559 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.692156_355.1 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:05.626+0000 7ff43592b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:05.627+0000 7ff43592b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:05Z" 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.565 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.566 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.567 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.569 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:05Z" 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.709706_360.1 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:10.738+0000 7f976202e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:10.739+0000 7f976202e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:10Z" 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.576 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:10Z" 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700428.214372_408.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700428.214372_408.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-04-10T00:01:38.581 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:38.582 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1018, in test_datalog_autotrim 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-35 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-35/ 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-35/ 2024-04-10T00:01:38.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-35/ 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:12:15 GMT 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:/xryqqk-35/ 2024-04-10T00:01:38.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EjHP2iSI7DNFh6ZZviM1wsdFRck= 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EjHP2iSI7DNFh6ZZviM1wsdFRck=', 'Content-Length': '0'} 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d7817a620d10f3de-006615bd3f-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-36 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-36/ 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-36/ 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-36/ 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:38.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:12:15 GMT 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:/xryqqk-36/ 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pRIoUbs0j2YAZy+AsNSWGhGp8L4= 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pRIoUbs0j2YAZy+AsNSWGhGp8L4=', 'Content-Length': '0'} 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000008f0a65482ad77d13-006615bd3f-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-35/ 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-35/ 2024-04-10T00:01:38.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-35/ 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.587 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:12:15 GMT 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:/xryqqk-35/ 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ZjW0F0f0t+FrfHdQpqDqHK3RhqQ= 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ZjW0F0f0t+FrfHdQpqDqHK3RhqQ=', 'Content-Length': '0'} 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000528a9af1c95bfb37-006615bd3f-1110-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-35/key 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-35/key 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-35/key 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:38.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:12:15 GMT 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:/xryqqk-35/key 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:J+72Y9FnecE4ZpOj4dGPbUVmthU= 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J+72Y9FnecE4ZpOj4dGPbUVmthU='} 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b79ad6834e8f78a-006615bd3f-1110-a1'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-36/ 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-36/ 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:38.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-36/ 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:12:15 GMT 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:/xryqqk-36/ 2024-04-10T00:01:38.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JxJ2RtGMEolNEnXlyu8TSswemgs= 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JxJ2RtGMEolNEnXlyu8TSswemgs=', 'Content-Length': '0'} 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b67345ae39fd3847-006615bd3f-1112-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-36/key 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-36/key 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-36/key 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:38.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:hBotaJrYa9FhFEdFPCLG/A== 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:12:15 GMT 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:/xryqqk-36/key 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JHKONE5O7m5J1LxinfBuqHT4lqs= 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:12:15 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JHKONE5O7m5J1LxinfBuqHT4lqs='} 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011c5211e82ee8231-006615bd3f-1112-a2'), ('Date', 'Tue, 09 Apr 2024 22:12:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700735.871719_444.1", 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:12:15.871719Z" 2024-04-10T00:01:38.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:16.101+0000 7fe717f83780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:16.102+0000 7fe717f83780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:16Z" 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.599 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.600 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.603 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:16Z" 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:21.208+0000 7f10e7bcf780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:21.209+0000 7f10e7bcf780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:21Z" 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.610 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:21Z" 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:26.322+0000 7fe297f42780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:26.323+0000 7fe297f42780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.892298_376.1", 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:26Z" 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.620 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.892298_376.1", 2024-04-10T00:01:38.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:26Z" 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.892298_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.892298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.892298_376.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.892298_376.1 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:31.524+0000 7f2be51e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:31.525+0000 7f2be51e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:31Z" 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.630 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:31Z" 2024-04-10T00:01:38.634 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:36.631+0000 7f6c66484780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:36.632+0000 7f6c66484780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:36Z" 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.640 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.642 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.643 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.644 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:36Z" 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:41.734+0000 7f95ab53c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:41.736+0000 7f95ab53c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:41Z" 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:38.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:41Z" 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.655 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:46.848+0000 7f73d5088780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:46.849+0000 7f73d5088780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:46Z" 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.661 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.662 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.664 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.665 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:46Z" 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:52.020+0000 7f3bb87d0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:52.021+0000 7f3bb87d0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:52Z" 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.672 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.673 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.676 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:52Z" 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:57.124+0000 7f650655e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:12:57.125+0000 7f650655e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:12:57Z" 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:12:57Z" 2024-04-10T00:01:38.688 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:02.233+0000 7fdc39f50780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:02.234+0000 7fdc39f50780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.693 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:02Z" 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.694 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.696 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.697 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.698 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:02Z" 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.699 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:07.344+0000 7fae7e183780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:07.345+0000 7fae7e183780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:07Z" 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.705 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.708 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:07Z" 2024-04-10T00:01:38.709 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:12.476+0000 7fefddcb5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:12.477+0000 7fefddcb5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:12Z" 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.715 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.718 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.719 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:12Z" 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:38.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:17.579+0000 7fb83a9ed780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:17.580+0000 7fb83a9ed780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:17Z" 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.726 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.727 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.728 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.729 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:17Z" 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.731 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:22.688+0000 7f45619e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:22.689+0000 7f45619e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:22Z" 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.737 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.739 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:22Z" 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.742 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:27.862+0000 7f9475f12780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:27.863+0000 7f9475f12780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:27Z" 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.748 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.749 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.750 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.752 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:27Z" 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.673463_351.1 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:32.970+0000 7fdb688d6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:32.971+0000 7fdb688d6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:32Z" 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:32Z" 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:38.080+0000 7ff862b89780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:38.081+0000 7ff862b89780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:38Z" 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.769 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.772 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.773 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:38Z" 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:43.184+0000 7f0fa1df3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:43.185+0000 7f0fa1df3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.694451_356.1", 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.694451Z", 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:43Z" 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.781 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.782 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.694451_356.1", 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.694451Z", 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:43Z" 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.694451_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.694451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.694451_356.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.694451_356.1 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:48.300+0000 7efd5ef73780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:48.301+0000 7efd5ef73780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:48Z" 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.792 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:48Z" 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:53.407+0000 7fb660878780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:53.408+0000 7fb660878780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:53Z" 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.801 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.802 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.803 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.805 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:53Z" 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:58.514+0000 7ff28bf8f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:13:58.515+0000 7ff28bf8f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:13:58Z" 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:13:58Z" 2024-04-10T00:01:38.816 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:03.700+0000 7f0127cc6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:03.701+0000 7f0127cc6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:03Z" 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.692156_355.1", 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.692156Z", 2024-04-10T00:01:38.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.826 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:03Z" 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.692156_355.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.692156Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.692156_355.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.692156_355.1 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:08.802+0000 7fb9140c4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:08.803+0000 7fb9140c4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:08Z" 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.833 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.834 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.836 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.837 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:08Z" 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:13.908+0000 7fdb475f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:13.909+0000 7fdb475f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:13Z" 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.843 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.847 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:13Z" 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:19.020+0000 7f8064bbf780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:19.021+0000 7f8064bbf780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:19Z" 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.855 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.856 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.857 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.858 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:19Z" 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:24.123+0000 7f3eb45b7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:24.124+0000 7f3eb45b7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:24Z" 2024-04-10T00:01:38.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.866 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:38.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:24Z" 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.869 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:29.227+0000 7fc2b0cff780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:29.228+0000 7fc2b0cff780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:29Z" 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.875 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.876 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.878 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.879 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:29Z" 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:34.335+0000 7fa3becf9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:34.336+0000 7fa3becf9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:34Z" 2024-04-10T00:01:38.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.889 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:34Z" 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.890 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:39.468+0000 7f7f3c0ac780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:39.471+0000 7f7f3c0ac780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:39Z" 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.898 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.899 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.900 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:39Z" 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:44.580+0000 7f81732f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:44.581+0000 7f81732f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:44Z" 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.906 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.910 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:44Z" 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:38.911 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:49.686+0000 7fe899004780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:49.687+0000 7fe899004780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:49Z" 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.917 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.920 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.921 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:49Z" 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:54.810+0000 7f6fe4a27780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:14:54.811+0000 7f6fe4a27780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:14:54Z" 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.928 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.929 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.930 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.931 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:14:54Z" 2024-04-10T00:01:38.932 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.933 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:00.009+0000 7fa63ca5c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:00.010+0000 7fa63ca5c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.694451_356.1", 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.694451Z", 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:00Z" 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.939 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.694451_356.1", 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.694451Z", 2024-04-10T00:01:38.942 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.943 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:00Z" 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.694451_356.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.694451Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.694451_356.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.694451_356.1 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:05.119+0000 7fd713d9a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:05.120+0000 7fd713d9a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:05Z" 2024-04-10T00:01:38.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.953 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:05Z" 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:10.239+0000 7f9f352b8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:10.240+0000 7f9f352b8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:10Z" 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.960 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.963 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:10Z" 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:15.460+0000 7f7b995e0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:15.461+0000 7f7b995e0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:15Z" 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.970 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.971 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.974 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:15Z" 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:20.566+0000 7fa731c36780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:20.567+0000 7fa731c36780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:38.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:20Z" 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:38.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.984 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:20Z" 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.756367_365.1 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:25.686+0000 7f774fb71780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:25.687+0000 7f774fb71780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:25Z" 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:38.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:38.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:25Z" 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:30.801+0000 7f8777de7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:30.803+0000 7f8777de7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:38.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:38.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:30Z" 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.001 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.004 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:30Z" 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:39.005 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:35.923+0000 7f788c7ff780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:35.924+0000 7f788c7ff780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:35Z" 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.011 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.012 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.013 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.014 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.015 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:35Z" 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.675822_352.1 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:41.039+0000 7f3f100bb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:41.040+0000 7f3f100bb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:41Z" 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.021 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.024 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.025 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:41Z" 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:46.169+0000 7fa14a3ec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:46.170+0000 7fa14a3ec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:46Z" 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.031 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.035 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:46Z" 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:51.298+0000 7f49fd94d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:51.299+0000 7f49fd94d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:51Z" 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.042 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:39.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:51Z" 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:56.482+0000 7f1e32530780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:15:56.483+0000 7f1e32530780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:15:56Z" 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.052 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.053 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:15:56Z" 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:01.586+0000 7f15d0f03780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:01.587+0000 7f15d0f03780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:01Z" 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:01Z" 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:06.709+0000 7fef4c2b8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:06.710+0000 7fef4c2b8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:06Z" 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.072 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.073 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:39.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.076 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:06Z" 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:11.949+0000 7f5819875780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:11.950+0000 7f5819875780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:11Z" 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.083 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.084 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.086 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.087 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:11Z" 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.675822_352.1 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:17.072+0000 7f974861c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:17.073+0000 7f974861c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:17Z" 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.096 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.098 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:17Z" 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:22.200+0000 7f33198f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:22.201+0000 7f33198f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:22Z" 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.104 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.105 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.106 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.108 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:22Z" 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:27.334+0000 7f2e63a2e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:27.335+0000 7f2e63a2e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:27Z" 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.115 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.117 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.663981_350.1", 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.663981Z", 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.118 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.119 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:27Z" 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.663981_350.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.663981Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.663981_350.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.663981_350.1 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:32.474+0000 7f512a860780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:32.476+0000 7f512a860780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:32Z" 2024-04-10T00:01:39.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.126 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:32Z" 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:37.599+0000 7f6bbc02b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:37.600+0000 7f6bbc02b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:37Z" 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.138 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.642202_346.1", 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.642202Z", 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:37Z" 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.642202_346.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.642202Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.642202_346.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.642202_346.1 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:42.728+0000 7fc924064780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:42.729+0000 7fc924064780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:42Z" 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.146 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.147 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.661401_349.1", 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.661401Z", 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:42Z" 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.661401_349.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.661401Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.661401_349.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.661401_349.1 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.151 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:47.846+0000 7f9a51973780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:47.847+0000 7f9a51973780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:47Z" 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.157 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.158 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.675822_352.1", 2024-04-10T00:01:39.159 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.675822Z", 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.160 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.161 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:47Z" 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.675822_352.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.675822Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.675822_352.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.675822_352.1 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:52.973+0000 7f6af6816780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:52.974+0000 7f6af6816780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:52Z" 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.168 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.169 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.704141_357.1", 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.704141Z", 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:52Z" 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.704141_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.704141Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.704141_357.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.704141_357.1 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:58.179+0000 7f0d1ee71780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:16:58.180+0000 7f0d1ee71780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:16:58Z" 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.178 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.180 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.651742_348.1", 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.651742Z", 2024-04-10T00:01:39.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:16:58Z" 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.651742_348.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.651742Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.651742_348.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.651742_348.1 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:03.311+0000 7f6a89d1d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:03.312+0000 7f6a89d1d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:17:03Z" 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.189 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.190 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.191 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.192 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.193 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:17:03Z" 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:39.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:08.448+0000 7f23a92c3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:08.449+0000 7f23a92c3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:17:08Z" 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.201 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.202 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.649397_347.1", 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.649397Z", 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:17:08Z" 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.649397_347.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.649397Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.649397_347.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.649397_347.1 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:13.579+0000 7fca5772d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:13.580+0000 7fca5772d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:39.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:17:13Z" 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.210 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.211 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.212 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.688829_353.1", 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.688829Z", 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.213 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.214 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:17:13Z" 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.688829_353.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.688829Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.688829_353.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.688829_353.1 2024-04-10T00:01:39.215 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:18.711+0000 7ffa8a505780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:17:18.712+0000 7ffa8a505780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:17:18Z" 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.221 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.222 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.223 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.673463_351.1", 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.673463Z", 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.224 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.225 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:17:18Z" 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.673463_351.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.673463Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700735.871719_444.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.673463_351.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712700735.871719_444.1 target=1_1712700424.673463_351.1 2024-04-10T00:01:39.226 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:39.227 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.227 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:39.227 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-04-10T00:01:39.227 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:39.228 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:39.228 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1061, in test_multi_zone_redirect 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:39.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a2", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "domain_root": "a2.rgw.meta:root", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "control_pool": "a2.rgw.control", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "gc_pool": "a2.rgw.log:gc", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "lc_pool": "a2.rgw.log:lc", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_pool": "a2.rgw.log", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "intent_log_pool": "a2.rgw.log:intent", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "usage_log_pool": "a2.rgw.log:usage", 2024-04-10T00:01:39.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "roles_pool": "a2.rgw.meta:roles", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "reshard_pool": "a2.rgw.log:reshard", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "otp_pool": "a2.rgw.otp", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "system_key": { 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "access_key": "BWDQKCT1041WUTW1", 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "secret_key": "0GPDcAr2gWAvBJ8BgqaDV4rQ6x9jAJ07" 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_pools": [ 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "default-placement", 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "index_pool": "a2.rgw.buckets.index", 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": { 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD": { 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "data_pool": "a2.rgw.buckets.data", 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compression_type": "random" 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "index_type": 0, 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "inline_data": true 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "notif_pool": "a2.rgw.log:notif" 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Sending period to new master zone a5723776-f142-428c-a6df-d7bf949257da 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "epoch": 5, 2024-04-10T00:01:39.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": [], 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_map": { 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zonegroups": [ 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a", 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a", 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": true, 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a1", 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:39.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a2", 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:39.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": false, 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:39.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:39.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [] 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "b", 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b", 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": false, 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:39.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b1", 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:39.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b2", 2024-04-10T00:01:39.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:39.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:39.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [] 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "short_zone_ids": [ 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2914326150 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 89109471 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 3383989928 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2084093761 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zonegroup": "a", 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_config": { 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_quota": { 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_quota": { 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:39.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_ratelimit": { 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_ratelimit": { 2024-04-10T00:01:39.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "anonymous_ratelimit": { 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_name": "test-realm", 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: Set sync_from_all flag on zone a2 to false 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-37 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-37/ 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-37/ 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-37/ 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:39.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:18:24 GMT 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:/xryqqk-37/ 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WmTusarOm4DJfzSD/pmFQtLa15k= 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:18:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WmTusarOm4DJfzSD/pmFQtLa15k=', 'Content-Length': '0'} 2024-04-10T00:01:39.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000150c08023d93ef76-006615beb0-1842-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:18:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-37/testredirect 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-37/testredirect 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-37/testredirect 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:3FCGuEcom6i4veFJuDiBdQ== 2024-04-10T00:01:39.251 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:18:24 GMT 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:/xryqqk-37/testredirect 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DKZ0Jpo0BLzz4GdQAGPSlP6LYlQ= 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:18:24 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DKZ0Jpo0BLzz4GdQAGPSlP6LYlQ='} 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003dd0ec11c73235e8-006615beb0-1842-a1'), ('Date', 'Tue, 09 Apr 2024 22:18:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:39.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:18:24.029085Z" 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:24.331+0000 7f1df689a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:24.332+0000 7f1df689a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.729688_362.1", 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:24Z" 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.259 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.261 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.729688_362.1", 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.262 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.263 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:24Z" 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.729688_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.729688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.729688_362.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.729688_362.1 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:29.445+0000 7fa2841da780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:29.446+0000 7fa2841da780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:39.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:29Z" 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.269 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.270 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.271 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:39.272 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.273 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:29Z" 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.756367_365.1 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:34.558+0000 7fc3fc5fb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:34.559+0000 7fc3fc5fb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.773250_367.1", 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:34Z" 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.280 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.281 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.773250_367.1", 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.284 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:34Z" 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.773250_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.773250Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.773250_367.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.773250_367.1 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:39.677+0000 7f0e985a7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:39.678+0000 7f0e985a7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.288 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:39Z" 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.290 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.292 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.293 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.294 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:39Z" 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.730558_363.1 2024-04-10T00:01:39.295 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:44.799+0000 7f6b98e93780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:44.800+0000 7f6b98e93780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.298 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.889732_375.1", 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.299 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:44Z" 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.301 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.302 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.303 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.889732_375.1", 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.304 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.305 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:44Z" 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:49.921+0000 7fe279640780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:49.922+0000 7fe279640780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700425.006471_386.1", 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:05.006471Z", 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:49Z" 2024-04-10T00:01:39.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.312 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.313 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700425.006471_386.1", 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.314 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:05.006471Z", 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:49Z" 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700425.006471_386.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:05.006471Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.316 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700425.006471_386.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700425.006471_386.1 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:55.031+0000 7f2198f29780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:18:55.032+0000 7f2198f29780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:18:55Z" 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.322 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.323 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.324 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.325 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.326 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:18:55Z" 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:00.155+0000 7faadc564780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:00.156+0000 7faadc564780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.949651_378.1", 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.949651Z", 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:00Z" 2024-04-10T00:01:39.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.334 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.949651_378.1", 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.949651Z", 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:00Z" 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.337 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.949651_378.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.949651Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.949651_378.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.949651_378.1 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:05.296+0000 7f596792a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:05.297+0000 7f596792a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:05Z" 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.343 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.344 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.345 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:39.346 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.347 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:05Z" 2024-04-10T00:01:39.348 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.350 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.730558_363.1 2024-04-10T00:01:39.350 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:10.423+0000 7f0c7f27b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:10.424+0000 7f0c7f27b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:10Z" 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.356 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.358 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:10Z" 2024-04-10T00:01:39.360 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:15.553+0000 7fbefcd9c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:15.554+0000 7fbefcd9c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.729688_362.1", 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-10T00:01:39.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:15Z" 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.366 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.729688_362.1", 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.370 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:15Z" 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.729688_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.729688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.729688_362.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.729688_362.1 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:20.686+0000 7f920df4b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:20.687+0000 7f920df4b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:20Z" 2024-04-10T00:01:39.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.377 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.378 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.380 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:20Z" 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:25.810+0000 7f6ab332f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:25.811+0000 7f6ab332f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:25Z" 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.387 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.730558_363.1", 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.730558Z", 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.390 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.391 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:25Z" 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.730558_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.730558Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.730558_363.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.730558_363.1 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:30.943+0000 7f2c5d97a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:30.944+0000 7f2c5d97a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:30Z" 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.398 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.399 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.401 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:30Z" 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:36.082+0000 7faa8a8df780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:36.083+0000 7faa8a8df780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:36Z" 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.408 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.409 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.411 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.412 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:36Z" 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:41.210+0000 7f8c7a590780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:41.211+0000 7f8c7a590780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.729688_362.1", 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:41Z" 2024-04-10T00:01:39.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.419 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.729688_362.1", 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.729688Z", 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.422 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:41Z" 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.729688_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.729688Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.729688_362.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.729688_362.1 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:46.355+0000 7f94d976f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:46.356+0000 7f94d976f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.426 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:46Z" 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.429 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.430 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.431 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.432 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:46Z" 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:51.486+0000 7fb9d35a7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:51.487+0000 7fb9d35a7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:51Z" 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.440 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.442 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.443 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:51Z" 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.445 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:56.628+0000 7f008b622780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:19:56.629+0000 7f008b622780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.448 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:19:56Z" 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.451 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.452 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.453 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.454 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:19:56Z" 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.455 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:01.801+0000 7f91b23f1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:01.802+0000 7f91b23f1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.458 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.459 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:01Z" 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.461 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.462 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.463 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.464 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.465 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:01Z" 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:06.932+0000 7f2d423ca780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:06.933+0000 7f2d423ca780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:06Z" 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.472 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.473 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.474 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.475 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:06Z" 2024-04-10T00:01:39.476 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:12.051+0000 7fa6a6231780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:12.052+0000 7fa6a6231780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:12Z" 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.484 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.485 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.486 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:12Z" 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:17.172+0000 7f6c9ed3b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:17.173+0000 7f6c9ed3b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:17Z" 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.492 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.493 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.495 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.496 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:17Z" 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:22.299+0000 7fb6ca9f6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:22.300+0000 7fb6ca9f6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.946918_377.1", 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:22Z" 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.504 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.946918_377.1", 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.506 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:22Z" 2024-04-10T00:01:39.507 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.946918_377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.946918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.946918_377.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.946918_377.1 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:27.423+0000 7f74d444d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:27.424+0000 7f74d444d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:27Z" 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:27Z" 2024-04-10T00:01:39.517 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:32.553+0000 7f256d707780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:32.554+0000 7f256d707780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.946918_377.1", 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:32Z" 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.523 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.946918_377.1", 2024-04-10T00:01:39.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.946918Z", 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:32Z" 2024-04-10T00:01:39.527 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.946918_377.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.946918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.946918_377.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.946918_377.1 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:37.675+0000 7f4789c7d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:37.676+0000 7f4789c7d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:37Z" 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.533 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.534 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.535 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.537 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:37Z" 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:42.805+0000 7f8706e74780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:42.806+0000 7f8706e74780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:42Z" 2024-04-10T00:01:39.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.544 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.545 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.709706_360.1", 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.546 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.709706Z", 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.547 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:42Z" 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.709706_360.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.709706Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.709706_360.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.709706_360.1 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:47.916+0000 7ffa909a7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:47.917+0000 7ffa909a7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.773250_367.1", 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:47Z" 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.558 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.773250_367.1", 2024-04-10T00:01:39.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.773250Z", 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:47Z" 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.773250_367.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.773250Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.561 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.773250_367.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.773250_367.1 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:53.041+0000 7fbcaec83780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:53.042+0000 7fbcaec83780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700426.099551_395.1", 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:06.099551Z", 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:53Z" 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.568 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700426.099551_395.1", 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:06.099551Z", 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.570 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.571 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:53Z" 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700426.099551_395.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:06.099551Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700426.099551_395.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700426.099551_395.1 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:58.179+0000 7fe271307780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:20:58.180+0000 7fe271307780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:20:58Z" 2024-04-10T00:01:39.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.727311_361.1", 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.727311Z", 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:20:58Z" 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.582 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.727311_361.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.727311Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.727311_361.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.727311_361.1 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:03.309+0000 7fb773522780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:03.310+0000 7fb773522780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:03Z" 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.588 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.589 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:39.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.592 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:03Z" 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.593 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.756367_365.1 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:08.451+0000 7fcf23af1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:08.452+0000 7fcf23af1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.732918_364.1", 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.732918Z", 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:08Z" 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.599 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.600 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.732918_364.1", 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.732918Z", 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.602 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:08Z" 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.732918_364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.732918Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.732918_364.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.732918_364.1 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:13.597+0000 7fa90ad79780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:13.598+0000 7fa90ad79780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.892298_376.1", 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:13Z" 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.609 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.610 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.892298_376.1", 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.892298Z", 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.613 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:13Z" 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.892298_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.892298Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.892298_376.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.892298_376.1 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:18.710+0000 7ff7bdf51780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:18.711+0000 7ff7bdf51780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:39.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:18Z" 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.620 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.621 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.756367_365.1", 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.756367Z", 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.622 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:18Z" 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.756367_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.756367Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.756367_365.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.756367_365.1 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:23.846+0000 7f9c57e10780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:23.847+0000 7f9c57e10780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:23Z" 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.630 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:23Z" 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:28.972+0000 7fa89bfa9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:28.974+0000 7fa89bfa9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:28Z" 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.641 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:28Z" 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:34.105+0000 7fec8e62c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:34.105+0000 7fec8e62c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:34Z" 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.650 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:34Z" 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.654 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:39.239+0000 7f5fc9b50780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:39.240+0000 7f5fc9b50780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:39Z" 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.661 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:39Z" 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.664 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:44.370+0000 7f78774f1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:44.371+0000 7f78774f1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.876862_374.1", 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-10T00:01:39.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:44Z" 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.670 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.671 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.672 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.876862_374.1", 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.673 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.674 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:44Z" 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:49.591+0000 7fa3f714e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:49.592+0000 7fa3f714e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:49Z" 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.681 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.682 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.683 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:39.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.685 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:49Z" 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:54.710+0000 7f533518c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:54.711+0000 7f533518c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.876862_374.1", 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:54Z" 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.692 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.694 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.876862_374.1", 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:54Z" 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:59.843+0000 7f2cf5b4e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:21:59.844+0000 7f2cf5b4e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:21:59Z" 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.703 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.707 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:21:59Z" 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:04.969+0000 7f2668003780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:04.970+0000 7f2668003780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.889732_375.1", 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:04Z" 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.714 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.715 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.889732_375.1", 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:04Z" 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:10.100+0000 7fbb57ade780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:10.101+0000 7fbb57ade780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.982479_383.1", 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.982479Z", 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:10Z" 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.725 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.726 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.727 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.982479_383.1", 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.982479Z", 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.728 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.729 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:10Z" 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.982479_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.982479Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.982479_383.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.982479_383.1 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:15.216+0000 7f7019403780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:15.217+0000 7f7019403780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.815777_372.1", 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.815777Z", 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:15Z" 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.736 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.738 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.815777_372.1", 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.815777Z", 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.739 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.740 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:15Z" 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.815777_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.815777Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.815777_372.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.815777_372.1 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:20.350+0000 7fcf27c5a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:20.351+0000 7fcf27c5a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:20Z" 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.747 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:20Z" 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.751 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:25.549+0000 7f02bea54780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:25.550+0000 7f02bea54780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:25Z" 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.757 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.758 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.759 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.762 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:25Z" 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:30.671+0000 7f64858ee780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:30.672+0000 7f64858ee780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.876862_374.1", 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:30Z" 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.770 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.876862_374.1", 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.876862Z", 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.772 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.773 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:30Z" 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.876862_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.876862Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.876862_374.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.876862_374.1 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:35.869+0000 7fd204f71780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:35.870+0000 7fd204f71780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:35Z" 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.780 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.781 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.782 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.874323_373.1", 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.874323Z", 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.783 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:35Z" 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.874323_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.874323Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.874323_373.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.874323_373.1 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:41.024+0000 7f56e5815780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:41.025+0000 7f56e5815780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:41Z" 2024-04-10T00:01:39.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.792 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:41Z" 2024-04-10T00:01:39.795 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:46.138+0000 7fcaac735780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:46.139+0000 7fcaac735780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.766836_366.1", 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.766836Z", 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:46Z" 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.801 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.802 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.766836_366.1", 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.766836Z", 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.804 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.805 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:46Z" 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.766836_366.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.766836Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.766836_366.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.766836_366.1 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:51.278+0000 7efe7ceb3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:51.279+0000 7efe7ceb3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:51Z" 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.814 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.815 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:51Z" 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.816 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:56.467+0000 7f962d8c9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:22:56.468+0000 7f962d8c9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:22:56Z" 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.824 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.825 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.826 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:22:56Z" 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:01.595+0000 7f05fc995780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:01.596+0000 7f05fc995780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.889732_375.1", 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-10T00:01:39.830 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.831 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:23:01Z" 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.832 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.834 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.889732_375.1", 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.889732Z", 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.835 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.836 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:23:01Z" 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.889732_375.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.889732Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.889732_375.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.889732_375.1 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:06.727+0000 7f845fcc9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:06.728+0000 7f845fcc9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.952757_380.1", 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.840 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.952757Z", 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.841 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:23:06Z" 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.842 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.844 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.952757_380.1", 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.952757Z", 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.845 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.846 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:23:06Z" 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.952757_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.952757Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.952757_380.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.952757_380.1 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:11.844+0000 7ff9bd753780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:11.845+0000 7ff9bd753780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:23:11Z" 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.852 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.854 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.855 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.856 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:23:11Z" 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:16.971+0000 7fc10ca47780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:16.972+0000 7fc10ca47780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:23:16Z" 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.862 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.864 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.780756_368.1", 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.780756Z", 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.865 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.866 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.867 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:23:16Z" 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.780756_368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.780756Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.780756_368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.780756_368.1 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:22.100+0000 7f12ec18c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:22.101+0000 7f12ec18c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:23:22Z" 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.873 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.813227_371.1", 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.813227Z", 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:23:22Z" 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.813227_371.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.813227Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.813227_371.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.813227_371.1 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:27.231+0000 7efedef47780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:23:27.232+0000 7efedef47780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:23:27Z" 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1712700424.798600_369.1", 2024-04-10T00:01:39.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:07:04.798600Z", 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:23:27Z" 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712700424.798600_369.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:07:04.798600Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712700424.798600_369.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701104.029085_507.1 target=1_1712700424.798600_369.1 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-04-10T00:01:39.889 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1173, in test_bucket_sync_disable 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zone_meta_checkpoint 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-46 2024-04-10T00:01:39.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-46/ 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-46/ 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-46/ 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.892 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:25:44 GMT 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:/xryqqk-46/ 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MxrmYYaRGkGRdxEx3GjDDNuuprk= 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MxrmYYaRGkGRdxEx3GjDDNuuprk=', 'Content-Length': '0'} 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000058cdd5c07fe2acd4-006615c068-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-47 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-47/ 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-47/ 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-47/ 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:39.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:25:44 GMT 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:/xryqqk-47/ 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:igcA+CdhSC69t/VPxDHOeyGG2Ow= 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:25:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:igcA+CdhSC69t/VPxDHOeyGG2Ow=', 'Content-Length': '0'} 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ad9bef747a9e9471-006615c068-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:39.894 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:39.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701544.676887_623.1", 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:25:44.676887Z" 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.896 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:25:44.917+0000 7f8a4955e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:25:44.918+0000 7f8a4955e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:25:44Z" 2024-04-10T00:01:39.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.902 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.903 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.904 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.905 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:25:44Z" 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:25:50.046+0000 7fb845c69780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:25:50.047+0000 7fb845c69780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:25:50Z" 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.912 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.913 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.914 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.915 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:25:50Z" 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:25:55.181+0000 7f98a5315780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:25:55.182+0000 7f98a5315780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:25:55Z" 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.922 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.923 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.924 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.925 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.926 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:25:55Z" 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:00.322+0000 7f71d9819780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:00.323+0000 7f71d9819780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:00Z" 2024-04-10T00:01:39.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.933 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.934 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.935 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.936 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.937 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:00Z" 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:05.437+0000 7f5551a2b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.938 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:05.438+0000 7f5551a2b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.939 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.940 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:05Z" 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.944 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.945 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.946 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.947 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:05Z" 2024-04-10T00:01:39.948 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:10.565+0000 7fa92e5bd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:10.566+0000 7fa92e5bd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:10Z" 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.954 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.956 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.957 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:10Z" 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:15.695+0000 7fe3d87a5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:15.696+0000 7fe3d87a5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:15Z" 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.965 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.966 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.967 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.968 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.969 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:15Z" 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:20.830+0000 7f3e63af9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:20.831+0000 7f3e63af9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:20Z" 2024-04-10T00:01:39.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.976 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.977 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.978 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:20Z" 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.980 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:25.960+0000 7f256e6de780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:25.961+0000 7f256e6de780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.982 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.983 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:25Z" 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.986 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.987 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.988 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.989 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.990 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:25Z" 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.991 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:31.086+0000 7f78ec131780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:31.087+0000 7f78ec131780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.992 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.993 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:39.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:39.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:31Z" 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.997 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.998 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:39.999 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.000 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.001 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:31Z" 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:36.211+0000 7fb81e600780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:36.212+0000 7fb81e600780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:36Z" 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.008 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.009 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.011 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.012 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:36Z" 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:41.351+0000 7f7ee9f56780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:41.352+0000 7f7ee9f56780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:41Z" 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.019 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.022 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:41Z" 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:46.474+0000 7fcc3e854780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:46.475+0000 7fcc3e854780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:46Z" 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.030 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.031 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.032 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.033 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.034 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:46Z" 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:51.601+0000 7f0dcfedb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:51.602+0000 7f0dcfedb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:51Z" 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.041 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.042 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.043 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:51Z" 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:56.743+0000 7fd25d769780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:26:56.744+0000 7fd25d769780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:26:56Z" 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.051 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.054 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.055 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:26:56Z" 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:01.865+0000 7fde9cad8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:01.866+0000 7fde9cad8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:01Z" 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.062 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:01Z" 2024-04-10T00:01:40.066 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:06.988+0000 7f742cbf7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:06.989+0000 7f742cbf7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:06Z" 2024-04-10T00:01:40.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.076 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.077 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:06Z" 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:27:11.260+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:12.108+0000 7fc79ca1e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:12.109+0000 7fc79ca1e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:12Z" 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.083 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.084 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.086 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.087 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:12Z" 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:17.242+0000 7f6e85c96780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:17.243+0000 7f6e85c96780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:17Z" 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.094 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.095 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.096 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:17Z" 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:22.365+0000 7f966938b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:22.367+0000 7f966938b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:22Z" 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.105 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.109 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:22Z" 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:27.487+0000 7fc84b776780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:27.488+0000 7fc84b776780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:27Z" 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.116 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.117 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.118 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.119 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.120 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:27Z" 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:32.625+0000 7f304a8c3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:32.626+0000 7f304a8c3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.123 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:32Z" 2024-04-10T00:01:40.126 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.127 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.129 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.130 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:32Z" 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:37.751+0000 7fdba9a67780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:37.752+0000 7fdba9a67780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:37Z" 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.138 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701104.029085_507.1", 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:18:24.029085Z", 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.140 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.141 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:37Z" 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701104.029085_507.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:18:24.029085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701104.029085_507.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701104.029085_507.1 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:42.900+0000 7f302c7a1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:42.902+0000 7f302c7a1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:42Z" 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.148 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.152 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:42Z" 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:48.019+0000 7f7ddfb5b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:48.020+0000 7f7ddfb5b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:48Z" 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.158 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.159 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.160 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:48Z" 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.265896_573.1 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:53.153+0000 7f4b6c52b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:53.154+0000 7f4b6c52b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:53Z" 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.170 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.172 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.173 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.174 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:53Z" 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.175 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.265896_573.1 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:58.314+0000 7fb021a15780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:27:58.315+0000 7fb021a15780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:27:58Z" 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.183 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:27:58Z" 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:03.429+0000 7fea058c4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:03.430+0000 7fea058c4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:03Z" 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.192 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.193 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.194 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.195 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.196 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:03Z" 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:08.560+0000 7f3e7aed7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:08.560+0000 7f3e7aed7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.201 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:08Z" 2024-04-10T00:01:40.202 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.205 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.206 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:08Z" 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.207 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:13.675+0000 7f0067dad780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:13.676+0000 7f0067dad780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:13Z" 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.214 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.217 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:13Z" 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.218 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:18.803+0000 7f32f94c8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:18.804+0000 7f32f94c8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:18Z" 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.224 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.225 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.226 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.227 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.228 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:18Z" 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.229 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:23.920+0000 7f7c7aabb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:23.921+0000 7f7c7aabb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.230 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.231 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.232 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.233 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.234 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:23Z" 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.235 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.236 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.237 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.238 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.239 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:23Z" 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.265896_573.1 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:29.029+0000 7fc436729780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:29.030+0000 7fc436729780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:29Z" 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.245 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.249 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:29Z" 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:34.176+0000 7f43b7f14780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:34.184+0000 7f43b7f14780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:34Z" 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.256 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.257 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.258 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.259 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.260 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:34Z" 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.261 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:39.383+0000 7fcb18ac3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:39.384+0000 7fcb18ac3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:39Z" 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.267 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.268 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.269 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.270 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:39Z" 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.271 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:44.513+0000 7fef9b881780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:44.514+0000 7fef9b881780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:44Z" 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.277 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.279 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:44Z" 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.282784_575.1 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:49.638+0000 7f8f2931c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:49.639+0000 7f8f2931c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.252615_571.1", 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:49Z" 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.287 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.252615_571.1", 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.290 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.291 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:49Z" 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.252615_571.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.252615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.252615_571.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.252615_571.1 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:54.761+0000 7fc258588780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:54.762+0000 7fc258588780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:54Z" 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.297 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.298 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.300 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.301 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:54Z" 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.207085_567.1 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:59.930+0000 7f4c54fdb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:28:59.931+0000 7f4c54fdb780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:28:59Z" 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.309 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.310 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.311 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:28:59Z" 2024-04-10T00:01:40.312 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:05.057+0000 7f9856219780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:05.058+0000 7f9856219780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.316 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.317 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:05Z" 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.318 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.320 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.321 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.322 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:05Z" 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:10.221+0000 7f534cdf2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:10.222+0000 7f534cdf2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.533935_583.1", 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.533935Z", 2024-04-10T00:01:40.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:10Z" 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.329 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.330 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.331 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.533935_583.1", 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.533935Z", 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.332 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:10Z" 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.333 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.533935_583.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.533935Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.533935_583.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.533935_583.1 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:15.331+0000 7f0771cb8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:15.332+0000 7f0771cb8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:15Z" 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.339 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.340 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.341 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.342 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.343 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:15Z" 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:20.471+0000 7f02c81a9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:20.472+0000 7f02c81a9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:20Z" 2024-04-10T00:01:40.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.350 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:20Z" 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.207085_567.1 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:25.654+0000 7f7cf1324780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:25.657+0000 7f7cf1324780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:25Z" 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.360 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.361 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.362 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.363 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:25Z" 2024-04-10T00:01:40.364 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.197429_564.1 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:30.808+0000 7f819d09c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:30.809+0000 7f819d09c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:30Z" 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.370 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.372 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.373 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.374 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:30Z" 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:35.932+0000 7ff945ca4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:35.934+0000 7ff945ca4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:35Z" 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.381 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.382 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.383 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.384 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.385 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:35Z" 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:41.053+0000 7f044e3f2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:41.055+0000 7f044e3f2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:41Z" 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.391 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.395 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:41Z" 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.396 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:46.174+0000 7f0c0eec0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:46.175+0000 7f0c0eec0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:46Z" 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.403 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.405 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.406 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:46Z" 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.207085_567.1 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:51.302+0000 7f666853d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:51.303+0000 7f666853d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:51Z" 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.414 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.415 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.417 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:51Z" 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:56.439+0000 7f66c2ef2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:29:56.440+0000 7f66c2ef2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:29:56Z" 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.423 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.426 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:29:56Z" 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.197429_564.1 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.428 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:01.557+0000 7f91278a5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:01.558+0000 7f91278a5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:01Z" 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.434 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.435 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:01Z" 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.438 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:06.703+0000 7f45491c2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:06.704+0000 7f45491c2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:06Z" 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:06Z" 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:11.848+0000 7f28799db780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:11.849+0000 7f28799db780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:11Z" 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.458 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.459 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.460 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.461 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:11Z" 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:16.990+0000 7f224d342780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:16.991+0000 7f224d342780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:16Z" 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.468 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.469 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.470 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.472 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:16Z" 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:22.114+0000 7f5f12bec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:22.115+0000 7f5f12bec780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:22Z" 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.479 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.480 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.481 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:22Z" 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.483 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:27.240+0000 7f4a51e3e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:27.241+0000 7f4a51e3e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:27Z" 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.489 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.490 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.491 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.492 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.493 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.494 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.495 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:27Z" 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:32.354+0000 7f0c06449780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:32.355+0000 7f0c06449780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.498 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.499 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:32Z" 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.501 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.502 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.503 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.504 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.505 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:32Z" 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.282784_575.1 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:37.478+0000 7f4490e6b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:37.479+0000 7f4490e6b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.509 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.528865_581.1", 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.510 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.511 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:37Z" 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.512 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.513 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.514 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.528865_581.1", 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.515 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.516 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:37Z" 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.528865_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.528865Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.528865_581.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.528865_581.1 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.517 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:42.603+0000 7fc5cf562780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:42.604+0000 7fc5cf562780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.520 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.521 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:42Z" 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.523 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.524 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.525 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.526 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:42Z" 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.528 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:47.738+0000 7fbd2aacc780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:47.739+0000 7fbd2aacc780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.529 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.530 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.531 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.532 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:47Z" 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.534 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.535 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.536 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.537 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:47Z" 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701544.676887_623.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701544.676887_623.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:40.539 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-04-10T00:01:40.540 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:40.540 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:40.540 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1200, in test_bucket_sync_enable_right_after_disable 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-48 2024-04-10T00:01:40.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/ 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/ 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/ 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.542 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:/xryqqk-48/ 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e6fThnabiwtLNmTDvuOuX8EpRjY= 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e6fThnabiwtLNmTDvuOuX8EpRjY=', 'Content-Length': '0'} 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000fc93aa1992be4ebb-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-49 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/ 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/ 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/ 2024-04-10T00:01:40.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:/xryqqk-49/ 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.544 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/AB5QdnZav1r8ZYK8pYyPEwkCyw= 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/AB5QdnZav1r8ZYK8pYyPEwkCyw=', 'Content-Length': '0'} 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000041615cde38abbbd4-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/ 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/ 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/ 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.545 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:/xryqqk-48/ 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef8c29b577dfb6d0-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/obj1 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/obj1 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/obj1 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:/xryqqk-48/obj1 2024-04-10T00:01:40.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:P1DjdPwCEYJlACFoi8tnxLgWz20= 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:P1DjdPwCEYJlACFoi8tnxLgWz20='} 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000170feee9d1dd26f2-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/ 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/ 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/ 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.548 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:/xryqqk-48/ 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4c4d30098e199f7-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/obj2 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/obj2 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/obj2 2024-04-10T00:01:40.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:/xryqqk-48/obj2 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.550 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tEQJB8u0lLSQi9U7D2Y9xQ81RMI= 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tEQJB8u0lLSQi9U7D2Y9xQ81RMI='} 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082d28bb78e5d9b46-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/ 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/ 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/ 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:/xryqqk-48/ 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000279924474fd57d6f-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/obj3 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/obj3 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/obj3 2024-04-10T00:01:40.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:/xryqqk-48/obj3 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:k+Mq83BqJMM77oh0eQQ9VeVMcXk= 2024-04-10T00:01:40.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k+Mq83BqJMM77oh0eQQ9VeVMcXk='} 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006c85b8d7cc42802-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/ 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/ 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/ 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.554 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:/xryqqk-48/ 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM= 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kpsiJGyMjSzOnM5iM13XGCEYBVM=', 'Content-Length': '0'} 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000204e1169497a67c8-006615c19c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-48/obj4 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-48/obj4 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-48/obj4 2024-04-10T00:01:40.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:/xryqqk-48/obj4 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:BoQymR+g1pd3wl0ne0wN+t6znAY= 2024-04-10T00:01:40.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BoQymR+g1pd3wl0ne0wN+t6znAY='} 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099f91dcb7c4428cd-006615c19c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/ 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/ 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/ 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.557 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:/xryqqk-49/ 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4= 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4=', 'Content-Length': '0'} 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8437be4abeb7ded-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/obj1 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/obj1 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/obj1 2024-04-10T00:01:40.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:/xryqqk-49/obj1 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ivGIpfw03YcGVBTAL+yTN8j9ZnY= 2024-04-10T00:01:40.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ivGIpfw03YcGVBTAL+yTN8j9ZnY='} 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ad4f99a054a248db-006615c19c-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/ 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/ 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/ 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.560 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:/xryqqk-49/ 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4= 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4=', 'Content-Length': '0'} 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f39161fbcf91ae-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/obj2 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/obj2 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/obj2 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:/xryqqk-49/obj2 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kbdIFZAoP2Fe14ypIproBHx8aeE= 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kbdIFZAoP2Fe14ypIproBHx8aeE='} 2024-04-10T00:01:40.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b16f48ce3c6f7a62-006615c19c-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/ 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/ 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/ 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.563 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:/xryqqk-49/ 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4= 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bLtYl48CslRN79zxAy9zioDKnd4=', 'Content-Length': '0'} 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e97aeb20e3dd9c2a-006615c19c-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/obj3 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/obj3 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/obj3 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:52 GMT 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:/xryqqk-49/obj3 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tFfLzVCn0FwBp1xtxQNsszCjjxk= 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:52 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tFfLzVCn0FwBp1xtxQNsszCjjxk='} 2024-04-10T00:01:40.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000284de68c837f36c0-006615c19c-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/ 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/ 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/ 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:40.566 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:53 GMT 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:/xryqqk-49/ 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JF3VNm/NdQ2sTI1n0ArNUy7UoOI= 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:30:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JF3VNm/NdQ2sTI1n0ArNUy7UoOI=', 'Content-Length': '0'} 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008486a311345f2964-006615c19d-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:30:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-49/obj4 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-49/obj4 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-49/obj4 2024-04-10T00:01:40.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:30:53 GMT 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:/xryqqk-49/obj4 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mWDL2SHiURmj6wZpauWVZHqoCTc= 2024-04-10T00:01:40.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:30:53 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mWDL2SHiURmj6wZpauWVZHqoCTc='} 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c75d366053c834bb-006615c19d-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:30:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701852.819902_651.1", 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:30:52.819902Z" 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:53.265+0000 7fc77c6d5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:53.266+0000 7fc77c6d5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701541.570209_607.1", 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:41.570209Z", 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:53Z" 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.577 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.578 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701541.570209_607.1", 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:41.570209Z", 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.580 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:53Z" 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.570209_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.570209Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.570209_607.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.570209_607.1 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:58.379+0000 7f65c1202780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:30:58.380+0000 7f65c1202780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:30:58Z" 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.587 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.589 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:30:58Z" 2024-04-10T00:01:40.591 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:03.491+0000 7f8147ac0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:03.492+0000 7f8147ac0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.252615_571.1", 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:03Z" 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.597 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.599 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.252615_571.1", 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.600 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.601 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:03Z" 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.252615_571.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.252615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.252615_571.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.252615_571.1 2024-04-10T00:01:40.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:08.619+0000 7f0a0075a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:08.620+0000 7f0a0075a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.604 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.605 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:08Z" 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.608 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.609 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.610 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.611 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.612 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:08Z" 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:13.733+0000 7fa627d47780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:13.734+0000 7fa627d47780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.615 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.270841_574.1", 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.618 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:13Z" 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.619 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.620 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.270841_574.1", 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.621 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.622 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:13Z" 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.270841_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.270841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.623 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.270841_574.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.270841_574.1 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:18.863+0000 7f6ef8c5f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:18.864+0000 7f6ef8c5f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.626 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.270841_574.1", 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-10T00:01:40.627 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.628 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:18Z" 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.629 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.630 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.631 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.270841_574.1", 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:18Z" 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.270841_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.270841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.270841_574.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.270841_574.1 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.634 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:24.045+0000 7f1e08677780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:24.046+0000 7f1e08677780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:24Z" 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.640 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.641 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.642 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.643 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:24Z" 2024-04-10T00:01:40.644 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:29.173+0000 7f2492404780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:29.174+0000 7f2492404780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.528865_581.1", 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:29Z" 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.650 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.651 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.652 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.528865_581.1", 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-10T00:01:40.653 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.654 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:29Z" 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.528865_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.528865Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.655 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.528865_581.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.528865_581.1 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:34.312+0000 7fdb130d1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:34.313+0000 7fdb130d1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.659 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.660 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:34Z" 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.661 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.662 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.663 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.664 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.665 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:34Z" 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:39.433+0000 7f7d1f550780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:39.434+0000 7f7d1f550780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:39Z" 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.671 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.672 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.673 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.674 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.675 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:39Z" 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.676 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.282784_575.1 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:44.543+0000 7f38b8483780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:44.544+0000 7f38b8483780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:44Z" 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.682 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.683 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:44Z" 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.265896_573.1 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:49.660+0000 7f8912ded780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:49.661+0000 7f8912ded780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:49Z" 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.692 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.694 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.695 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.696 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:49Z" 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.697 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.207085_567.1 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:54.805+0000 7f366cb5d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:54.806+0000 7f366cb5d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.702 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:54Z" 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.703 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.704 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.705 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.706 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:54Z" 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.707 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.265896_573.1 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:59.939+0000 7fd32a9d4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:31:59.940+0000 7fd32a9d4780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701541.011327_591.1", 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:41.011327Z", 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.711 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.712 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:31:59Z" 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.713 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.714 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.715 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.716 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701541.011327_591.1", 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:41.011327Z", 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.717 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:31:59Z" 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.718 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.011327_591.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.011327Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.011327_591.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.011327_591.1 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:05.063+0000 7f75701ab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:05.064+0000 7f75701ab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.722 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.723 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:05Z" 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.724 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.725 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.726 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.727 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.728 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:05Z" 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:10.207+0000 7f2b3574b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:10.208+0000 7f2b3574b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.731 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.732 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701541.430664_598.1", 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:41.430664Z", 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.733 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:10Z" 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.735 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.736 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701541.430664_598.1", 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:41.430664Z", 2024-04-10T00:01:40.737 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.738 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:10Z" 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.430664_598.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.430664Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.430664_598.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.430664_598.1 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:32:12.317+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:15.331+0000 7fddf0202780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:15.332+0000 7fddf0202780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.740 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.741 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.742 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.743 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:15Z" 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.745 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.746 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.747 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.265896_573.1", 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.265896Z", 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.748 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.749 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:15Z" 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.265896_573.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.265896Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.265896_573.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.265896_573.1 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:20.459+0000 7f0900d53780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:20.460+0000 7f0900d53780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701543.601335_611.1", 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:43.601335Z", 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:20Z" 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.755 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.756 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.757 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701543.601335_611.1", 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.758 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:43.601335Z", 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.759 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:20Z" 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701543.601335_611.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:43.601335Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701543.601335_611.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701543.601335_611.1 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:25.579+0000 7ffaed00b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:25.580+0000 7ffaed00b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.762 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.763 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701541.012414_592.1", 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:41.012414Z", 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:25Z" 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.766 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.767 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.768 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701541.012414_592.1", 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:41.012414Z", 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.769 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:25Z" 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.012414_592.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.012414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.012414_592.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.012414_592.1 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:30.713+0000 7fa83c982780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.771 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:30.714+0000 7fa83c982780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.773 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:30Z" 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.776 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.777 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.778 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.779 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.780 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:30Z" 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.781 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:35.826+0000 7f4b904ed780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:35.827+0000 7f4b904ed780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.782 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.783 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.784 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.785 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:35Z" 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.787 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.788 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.789 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.790 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.791 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:35Z" 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:40.946+0000 7fb34d280780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:40.947+0000 7fb34d280780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.793 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.794 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:40Z" 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.798 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.799 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.800 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.801 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:40Z" 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.802 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.207085_567.1 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:46.083+0000 7f61ea2e5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:46.085+0000 7f61ea2e5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.803 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.804 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:46Z" 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.808 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.809 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.810 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.811 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:46Z" 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.813 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:51.224+0000 7f6b0a6b7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:51.225+0000 7f6b0a6b7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:51Z" 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.819 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.820 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:51Z" 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:56.352+0000 7f44d0a18780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:32:56.353+0000 7f44d0a18780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:32:56Z" 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.830 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.832 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.833 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:32:56Z" 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.834 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:01.470+0000 7f7f46667780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:01.471+0000 7f7f46667780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:01Z" 2024-04-10T00:01:40.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.840 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.842 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:01Z" 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:06.610+0000 7f25cfbc1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:06.611+0000 7f25cfbc1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:06Z" 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.850 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.852 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.853 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.854 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:06Z" 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.855 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:11.728+0000 7f69f6b8c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:11.729+0000 7f69f6b8c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:11Z" 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.861 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.862 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.863 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.282784_575.1", 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.282784Z", 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.864 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.865 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:11Z" 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.282784_575.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.282784Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.282784_575.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.282784_575.1 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:16.860+0000 7f312e852780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:16.861+0000 7f312e852780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.869 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.870 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:16Z" 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.872 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.873 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.874 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.207085_567.1", 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.207085Z", 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.875 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.876 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:16Z" 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.207085_567.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.207085Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.207085_567.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.207085_567.1 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.877 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:22.001+0000 7f931a97f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:22.002+0000 7f931a97f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:22Z" 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.883 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.884 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.885 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.886 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.887 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:22Z" 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:27.202+0000 7f8cf5bf0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:27.203+0000 7f8cf5bf0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:27Z" 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.894 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.895 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.896 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.897 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:27Z" 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.898 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:32.341+0000 7f27829a8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:32.342+0000 7f27829a8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701541.012414_592.1", 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:41.012414Z", 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:32Z" 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.904 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.905 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.906 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701541.012414_592.1", 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:41.012414Z", 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.907 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.908 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:32Z" 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.012414_592.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.012414Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.012414_592.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.909 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.012414_592.1 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:37.485+0000 7f98609b6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:37.486+0000 7f98609b6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.528865_581.1", 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:37Z" 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.916 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.917 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.528865_581.1", 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.528865Z", 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.918 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.919 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:37Z" 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.528865_581.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.528865Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.528865_581.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.528865_581.1 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:42.662+0000 7fac2c8d3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:42.663+0000 7fac2c8d3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701541.008529_590.1", 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:41.008529Z", 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:42Z" 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.926 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.927 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701541.008529_590.1", 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:41.008529Z", 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.930 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:42Z" 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701541.008529_590.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:41.008529Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701541.008529_590.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701541.008529_590.1 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:47.794+0000 7f16abd40780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:47.795+0000 7f16abd40780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.252615_571.1", 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:47Z" 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.936 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.938 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.252615_571.1", 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.252615Z", 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.939 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:47Z" 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.252615_571.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.252615Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.252615_571.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.252615_571.1 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.941 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:52.916+0000 7f0a167f6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:52.917+0000 7f0a167f6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.270841_574.1", 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:52Z" 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.947 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.948 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.270841_574.1", 2024-04-10T00:01:40.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.270841Z", 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:52Z" 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.951 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.270841_574.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.270841Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.270841_574.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.270841_574.1 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:58.063+0000 7f14ddc6e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:33:58.064+0000 7f14ddc6e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:33:58Z" 2024-04-10T00:01:40.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.958 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:33:58Z" 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.962 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:03.196+0000 7efee1756780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:03.197+0000 7efee1756780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:03Z" 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.968 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.969 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.972 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:03Z" 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:08.330+0000 7f221e9dd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:08.331+0000 7f221e9dd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:08Z" 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.978 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.197429_564.1", 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.197429Z", 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.981 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.982 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:08Z" 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.197429_564.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.197429Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.197429_564.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.197429_564.1 2024-04-10T00:01:40.983 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:13.478+0000 7f49ba6c7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:13.479+0000 7f49ba6c7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.984 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:13Z" 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.989 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.990 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.991 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.202102_565.1", 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.202102Z", 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:40.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:40.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:13Z" 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.202102_565.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.202102Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.202102_565.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.202102_565.1 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:40.994 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:18.592+0000 7fd5e57a3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:18.593+0000 7fd5e57a3780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.995 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:40.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:40.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:40.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:40.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:40.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:18Z" 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.001 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:18Z" 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.005 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:23.722+0000 7f54ff5e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:23.723+0000 7f54ff5e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.530215_582.1", 2024-04-10T00:01:41.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.530215Z", 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:23Z" 2024-04-10T00:01:41.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.012 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.530215_582.1", 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.530215Z", 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.015 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:23Z" 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.016 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.530215_582.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.530215Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.530215_582.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.530215_582.1 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:28.853+0000 7fbada41e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:28.854+0000 7fbada41e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:41.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.021 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:28Z" 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.022 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.024 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.524385_580.1", 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.524385Z", 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.025 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.026 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:28Z" 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.524385_580.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.524385Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.524385_580.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.524385_580.1 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:33.981+0000 7f0673417780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:33.983+0000 7f0673417780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:33Z" 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.034 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.035 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701540.902795_586.1", 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:25:40.902795Z", 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.037 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:33Z" 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701540.902795_586.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:25:40.902795Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701540.902795_586.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712701852.819902_651.1 target=1_1712701540.902795_586.1 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:39.111+0000 7fb634d18780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:39.112+0000 7fb634d18780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.039 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.040 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712701852.819902_651.1", 2024-04-10T00:01:41.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:30:52.819902Z", 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:34:39Z" 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.044 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.045 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "marker": "1_1712701852.819902_651.1", 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:30:52.819902Z", 2024-04-10T00:01:41.046 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.047 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:34:39Z" 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712701852.819902_651.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:30:52.819902Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712701852.819902_651.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-48 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-48 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-48 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:39.368+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:44.369+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:49.369+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:54.370+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:34:59.370+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:04.371+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:09.371+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:14.372+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:19.373+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:24.374+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.050 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:29.374+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:34.374+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:39.375+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:44.375+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:49.376+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:54.376+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:35:59.377+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:04.377+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:09.378+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:14.379+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:19.379+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:24.380+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:29.380+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:34.381+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.051 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:39.381+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:44.382+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:49.382+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:54.383+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:36:59.383+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:04.385+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:09.386+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:37:12.320+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:14.385+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:19.386+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:24.386+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:29.387+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:34.388+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:39.388+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.052 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:44.389+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:49.389+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:54.390+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:37:59.391+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:04.391+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:09.392+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:14.393+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:19.393+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:24.394+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:29.394+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:34.395+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:39.396+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:44.396+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:49.397+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.053 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:54.398+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:38:59.397+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:04.398+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:09.398+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:14.399+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:19.400+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:24.400+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:29.401+0000 7fe5a5201780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:34.402+0000 7fe5a5201780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:34.402+0000 7fe5a5201780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:34.402+0000 7fe5a5201780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.054 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.055 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1235, in test_bucket_sync_disable_enable 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket_name) 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-50 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/ 2024-04-10T00:01:41.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/ 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/ 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.057 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:/xryqqk-50/ 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qLg2V09L/CpBoOqDvoNHgxsW0Ow= 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qLg2V09L/CpBoOqDvoNHgxsW0Ow=', 'Content-Length': '0'} 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b8f8d04aca85619a-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-51 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/ 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/ 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/ 2024-04-10T00:01:41.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:/xryqqk-51/ 2024-04-10T00:01:41.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:aat8SuVMcnyF3RBB8x+QcjdaBs8= 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:aat8SuVMcnyF3RBB8x+QcjdaBs8=', 'Content-Length': '0'} 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f1542495d10f1f09-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/ 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/ 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/ 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:/xryqqk-50/ 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014dd47594379c36c-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/obj1 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/obj1 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/obj1 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:/xryqqk-50/obj1 2024-04-10T00:01:41.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6MdLtm6kX8l+pAHlRu59mJIkbUA= 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6MdLtm6kX8l+pAHlRu59mJIkbUA='} 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2c808c5aa52f1df-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/ 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/ 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/ 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:/xryqqk-50/ 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000295e113317febf78-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/obj2 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/obj2 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/obj2 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:/xryqqk-50/obj2 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.065 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SRoamgSHg3ApEoidjYp3o4yjQTA= 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SRoamgSHg3ApEoidjYp3o4yjQTA='} 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d42a2d2ee965fd6-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/ 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/ 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/ 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:/xryqqk-50/ 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae367eb33e98f8a0-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/obj3 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/obj3 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/obj3 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:/xryqqk-50/obj3 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.068 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6FYHZNZ68U1RgjxCIdHYikQbMx0= 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6FYHZNZ68U1RgjxCIdHYikQbMx0='} 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000577fa1a71a9e8f9e-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/ 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/ 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/ 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:/xryqqk-50/ 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc= 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:JwECF1h6q+f521NbY0PIKFhZDNc=', 'Content-Length': '0'} 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a87e8d2976d50cc-006615c3a6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-50/obj4 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-50/obj4 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-50/obj4 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:/xryqqk-50/obj4 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.071 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MlepQsXi+A3qSyIkeO1WvpdbEuU= 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MlepQsXi+A3qSyIkeO1WvpdbEuU='} 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004294a534cf82e990-006615c3a6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/ 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/ 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/ 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:/xryqqk-51/ 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f6c15b5045b9fda-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/obj1 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/obj1 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/obj1 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:/xryqqk-51/obj1 2024-04-10T00:01:41.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:CBDc+h/znwMESDV1oGTGMTuKm98= 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:CBDc+h/znwMESDV1oGTGMTuKm98='} 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8972415332b3fd7-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/ 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/ 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/ 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:/xryqqk-51/ 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008b6ceb0909c5b1b-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/obj2 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/obj2 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/obj2 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:/xryqqk-51/obj2 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.077 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Dxqb22Cfcr3OnszFBd//4JTGOHg= 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Dxqb22Cfcr3OnszFBd//4JTGOHg='} 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a961fca81ab338c-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/ 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/ 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/ 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:/xryqqk-51/ 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075ac6d8fbfea1c92-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/obj3 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/obj3 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/obj3 2024-04-10T00:01:41.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:/xryqqk-51/obj3 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EsyQmDPQTCksxSIJbr2LAQ07r98= 2024-04-10T00:01:41.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EsyQmDPQTCksxSIJbr2LAQ07r98='} 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c693319638ef7388-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/ 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/ 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/ 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.081 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:/xryqqk-51/ 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M= 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7mnSE094IL5lA7F6sEu+8cMsv+M=', 'Content-Length': '0'} 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a659987deefb882-006615c3a6-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-51/obj4 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-51/obj4 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-51/obj4 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:39:34 GMT 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:/xryqqk-51/obj4 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:E+a60A2n/maFBNnPkSe5ejj5u9w= 2024-04-10T00:01:41.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:39:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:E+a60A2n/maFBNnPkSe5ejj5u9w='} 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6bfaddf696469bf-006615c3a6-1b6f-a2'), ('Date', 'Tue, 09 Apr 2024 22:39:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702374.479594_726.1", 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:39:34.479594Z" 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.085 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:34.894+0000 7fa67c131780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:34.895+0000 7fa67c131780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702374.442860_719.1", 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:39:34.442860Z", 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:39:34Z" 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.091 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.092 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.093 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "marker": "1_1712702374.442860_719.1", 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:39:34.442860Z", 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.094 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.095 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:39:34Z" 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702374.442860_719.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:39:34.442860Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.442860_719.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712702374.479594_726.1 target=1_1712702374.442860_719.1 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:40.029+0000 7fd14073e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:40.030+0000 7fd14073e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.097 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702374.479594_726.1", 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:39:34.479594Z", 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:39:40Z" 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.101 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.102 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.103 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "marker": "1_1712702374.479594_726.1", 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.104 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:39:34.479594Z", 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.105 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:39:40Z" 2024-04-10T00:01:41.106 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702374.479594_726.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:39:34.479594Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702374.479594_726.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-50 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-50 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-50 2024-04-10T00:01:41.107 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:40.283+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:45.284+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:50.284+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:39:55.285+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:00.286+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:05.286+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:10.287+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:15.288+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.108 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:20.288+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:25.289+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:30.290+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:35.291+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:40.291+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:45.291+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:50.292+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:40:55.292+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:00.293+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:05.293+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:10.294+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:15.294+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:20.295+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:25.295+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:30.296+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.109 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:35.296+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:40.297+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:45.297+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:50.298+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:41:55.298+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:00.299+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:05.300+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:10.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:42:13.382+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:15.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:20.300+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:25.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:30.301+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.110 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:35.302+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:40.303+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:45.303+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:50.304+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:42:55.304+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:00.305+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:05.305+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:10.306+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:15.307+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:20.308+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:25.308+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:30.309+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:35.309+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:40.310+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:45.310+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.111 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:50.311+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:43:55.312+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:00.313+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:05.313+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:10.313+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:15.314+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:20.314+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:25.315+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:30.315+0000 7f19aef4d780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:35.316+0000 7f19aef4d780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:35.316+0000 7f19aef4d780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:35.316+0000 7f19aef4d780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.112 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1277, in test_multipart_object_sync 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.113 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-52 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/ 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/ 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/ 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:35 GMT 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:/xryqqk-52/ 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:jIT6FJ1sbYUczUOjGoZTnMO9RuE= 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jIT6FJ1sbYUczUOjGoZTnMO9RuE=', 'Content-Length': '0'} 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004d054849f78618fd-006615c4d3-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:44:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-53 2024-04-10T00:01:41.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-53/ 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-53/ 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-53/ 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.116 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:35 GMT 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:/xryqqk-53/ 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WJeUinx6DvYGq+3nZqYXjAPYwhI= 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WJeUinx6DvYGq+3nZqYXjAPYwhI=', 'Content-Length': '0'} 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b6f52863568b601e-006615c4d3-1b6f-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:44:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploads 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploads 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploads 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:POST 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:35 GMT 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?uploads 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:fxulVOH4uDLtWqBpeag1DuxrBgI= 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:fxulVOH4uDLtWqBpeag1DuxrBgI=', 'Content-Length': '0'} 2024-04-10T00:01:41.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009893d3443fd98759-006615c4d3-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'xryqqk-52MULTIPART2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW' 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=1 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=1 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=1 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.119 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:ebKBBg0ze5srhMzzkK3PdA== 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:37 GMT 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?partNumber=1&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cq6IDxefIVFXta/yfS49/BGflD4= 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cq6IDxefIVFXta/yfS49/BGflD4='} 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc63dc1f8e85d646-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=2 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=2 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=2 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:dIQ6OrGTo4m87YmUAtmdXw== 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:37 GMT 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?partNumber=2&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7w8CLDcI67b2TbeHFiOZ1UPA/Z0= 2024-04-10T00:01:41.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7w8CLDcI67b2TbeHFiOZ1UPA/Z0='} 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fe10f8041fd8e95-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=3 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=3 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=3 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:e4RW4edMN49Fhh9TYZ51tg== 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:37 GMT 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?partNumber=3&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:J3cVPJ/AP1WpT1wberONa1SWw0Q= 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J3cVPJ/AP1WpT1wberONa1SWw0Q='} 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3ded15ac38d7f1e-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=4 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=4 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW&partNumber=4 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:l+jDi4yPcIlVvEiQwDQZTQ== 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:37 GMT 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?partNumber=4&uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EY3ZhnzyF8+CkAOoBP4mW54eZx0= 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EY3ZhnzyF8+CkAOoBP4mW54eZx0='} 2024-04-10T00:01:41.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a80e22f7cfffca7-006615c4d5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:41.125 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:37 GMT 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2jDxbPYwds8+T3ijCTLRTnzylj4= 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2jDxbPYwds8+T3ijCTLRTnzylj4=', 'Content-Length': '0'} 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000984b0b3a6c72962a-006615c4d5-1b6d-a1'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: POST 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 1 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: "79b281060d337b9b2b84ccf390adcf74" 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: "74843a3ab193a389bced899402d99d5f" 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 3 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: "7b8456e1e74c378f45861f53619e75b6" 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.127 INFO:tasks.rgw_multisite_tests: 4 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests: "97e8c38b8c8f708955bc4890c034194d" 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'Content-Type': 'text/xml'} 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:POST 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:text/xml 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:44:37 GMT 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:/xryqqk-52/MULTIPART?uploadId=2~9_OvIzedLu3f7cz0pgF_oOiq1mX8koW 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.128 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uLluuPnzl2KeAiIPGZJxSknqK2U= 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:44:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uLluuPnzl2KeAiIPGZJxSknqK2U=', 'Content-Length': '460'} 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f5fd54acbd92cf77-006615c4d5-1b6d-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Tue, 09 Apr 2024 22:44:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: http://smithi063.front.sepia.ceph.com:8000/xryqqk-52/MULTIPARTxryqqk-52MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.129 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702675.402940_775.1", 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:44:35.402940Z" 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702675.402940_775.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:37.571+0000 7f5ecad85780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:37.572+0000 7f5ecad85780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702675.402940_775.1", 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:44:35.402940Z", 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:44:37Z" 2024-04-10T00:01:41.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.136 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "marker": "1_1712702675.402940_775.1", 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:44:35.402940Z", 2024-04-10T00:01:41.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:44:37Z" 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702675.402940_775.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:44:35.402940Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702675.402940_775.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702675.402940_775.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.140 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-52 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-52 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-52 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-52 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:37.836+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:42.837+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:47.837+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:52.838+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:44:57.839+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:02.839+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:07.840+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:12.840+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:17.840+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:22.841+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:27.842+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:32.843+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:37.844+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:42.843+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:47.844+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:52.845+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:45:57.845+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:02.846+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:07.846+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:12.847+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:17.847+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:22.848+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:27.848+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:32.849+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:37.849+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:42.850+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:47.851+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:52.851+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:46:57.852+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:02.853+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:07.853+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:12.854+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:17.855+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:22.854+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:27.855+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:32.855+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:47:33.387+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:37.856+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:42.856+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:47.857+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:52.857+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:47:57.858+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:02.858+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:07.859+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:12.860+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:17.860+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:22.861+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:27.861+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:32.862+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:37.863+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:42.863+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:47.864+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:52.865+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:48:57.866+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:02.865+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:07.866+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:12.866+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:17.867+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:22.867+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:27.868+0000 7f5e4d9d1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:32.868+0000 7f5e4d9d1780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:32.868+0000 7f5e4d9d1780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:32.868+0000 7f5e4d9d1780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_index_log_trim 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.146 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1343, in test_bucket_index_log_trim 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1339, in make_test_bucket 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-55 2024-04-10T00:01:41.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/ 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/ 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/ 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.148 INFO:tasks.rgw_multisite_tests:/xryqqk-55/ 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7AablSRZxA0ofierpiO6O4CcbBM= 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7AablSRZxA0ofierpiO6O4CcbBM=', 'Content-Length': '0'} 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000152cb456ff34fd84-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/ 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/ 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/ 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:/xryqqk-55/ 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8bf2e4adbf6df6f-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/a 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/a 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/a 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:/xryqqk-55/a 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:dsLHuDD2EEfCSKN3tiv1cZtLxXE= 2024-04-10T00:01:41.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dsLHuDD2EEfCSKN3tiv1cZtLxXE='} 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bccf8cc5490a9073-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/ 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/ 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/ 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.152 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:/xryqqk-55/ 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000108d6f58a8dccbb5-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/b 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/b 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/b 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:/xryqqk-55/b 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QaB7ldOJnu2/yWFh268AA/8hL8Y= 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QaB7ldOJnu2/yWFh268AA/8hL8Y='} 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a6597b6be01f3a1-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/ 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/ 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/ 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.155 INFO:tasks.rgw_multisite_tests:/xryqqk-55/ 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2d9d387ea392614-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/c 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/c 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/c 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:/xryqqk-55/c 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:jluzeo3BD4LQtpkTkt12cs/xr4U= 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:jluzeo3BD4LQtpkTkt12cs/xr4U='} 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c05f02a439d9a97-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/ 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/ 2024-04-10T00:01:41.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/ 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.158 INFO:tasks.rgw_multisite_tests:/xryqqk-55/ 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs= 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:c9Bc0wiR4XxWHRtFHeHius3uWTs=', 'Content-Length': '0'} 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081b23efab3c2b47c-006615c5fd-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-55/d 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-55/d 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-55/d 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:49:33 GMT 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:/xryqqk-55/d 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:31dbqPlxGsL+JI3cOAgB9dxiJIQ= 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:49:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:31dbqPlxGsL+JI3cOAgB9dxiJIQ='} 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa0f3f5295d357fa-006615c5fd-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:49:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702973.352006_826.1", 2024-04-10T00:01:41.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:49:33.352006Z" 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:33.690+0000 7fe5e6021780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:33.691+0000 7fe5e6021780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702973.335445_823.1", 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:49:33.335445Z", 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:49:33Z" 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.167 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.168 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "marker": "1_1712702973.335445_823.1", 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.170 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:49:33.335445Z", 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:49:33Z" 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.172 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702973.335445_823.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:49:33.335445Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.335445_823.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712702973.352006_826.1 target=1_1712702973.335445_823.1 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:38.822+0000 7fc1734a0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:38.823+0000 7fc1734a0780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712702973.352006_826.1", 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:49:33.352006Z", 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:49:38Z" 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.178 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.179 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.180 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "marker": "1_1712702973.352006_826.1", 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:49:33.352006Z", 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.181 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.182 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:49:38Z" 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712702973.352006_826.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:49:33.352006Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712702973.352006_826.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-55 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.183 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-55 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-55 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:39.076+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:44.077+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.184 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:49.077+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:54.078+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:49:59.079+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:04.079+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:09.080+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:14.080+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:19.081+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:24.082+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:29.082+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:34.083+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:39.084+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:44.084+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:49.084+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.185 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:54.085+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:50:59.086+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:04.086+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:09.086+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:14.087+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:19.088+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:24.088+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:29.089+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:34.089+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:39.090+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:44.090+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:49.091+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:54.091+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.186 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:51:59.092+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:04.092+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:09.093+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:14.094+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:19.095+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:24.094+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:29.095+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:52:33.390+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:34.095+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:39.096+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:44.096+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:49.096+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:54.097+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.187 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:52:59.098+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:04.099+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:09.099+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:14.100+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:19.101+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:24.102+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:29.102+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:34.103+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:39.103+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:44.104+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:49.105+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:54.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:53:59.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:04.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:09.106+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:14.107+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:19.108+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:24.108+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:29.109+0000 7ff6276da780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:34.110+0000 7ff6276da780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:34.110+0000 7ff6276da780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:34.110+0000 7ff6276da780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-04-10T00:01:41.189 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1397, in test_bucket_reshard_index_log_trim 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: test_bucket = make_test_bucket() 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1393, in make_test_bucket 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, name) 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.190 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-56 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/ 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/ 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/ 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:/xryqqk-56/ 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uVaePOb/pA0dF1rNaoJQ9MlENG0= 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uVaePOb/pA0dF1rNaoJQ9MlENG0=', 'Content-Length': '0'} 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000049d5522180b23483-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/ 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/ 2024-04-10T00:01:41.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/ 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:/xryqqk-56/ 2024-04-10T00:01:41.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efdd6fbe26344f18-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/a 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/a 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/a 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:/xryqqk-56/a 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YXHIrAS4GLSjb6FGhFDXbq/BNPI= 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YXHIrAS4GLSjb6FGhFDXbq/BNPI='} 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbf7aa6511cbffd6-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/ 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/ 2024-04-10T00:01:41.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/ 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:/xryqqk-56/ 2024-04-10T00:01:41.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be35ba9afa081219-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/b 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/b 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/b 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:/xryqqk-56/b 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XFXuCeJm1ROoJHqR8M7VY7il4NM= 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XFXuCeJm1ROoJHqR8M7VY7il4NM='} 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000537b62a3452589fd-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/ 2024-04-10T00:01:41.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/ 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/ 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:/xryqqk-56/ 2024-04-10T00:01:41.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002152c1b57b0c8471-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/c 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/c 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/c 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:/xryqqk-56/c 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zySh4Lh4wlYBKDuXrjl8WexxW5s= 2024-04-10T00:01:41.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zySh4Lh4wlYBKDuXrjl8WexxW5s='} 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbd73fa971cef15c-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/ 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/ 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/ 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.203 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:/xryqqk-56/ 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4= 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:L/bkZBAACBhJCfSOl5rosVCOTB4=', 'Content-Length': '0'} 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac73ead0aaad4fd9-006615c72a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-56/d 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-56/d 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-56/d 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.205 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:54:34 GMT 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:/xryqqk-56/d 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tGTuH680e5BMsuq0tQpLbFRbnD8= 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:54:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tGTuH680e5BMsuq0tQpLbFRbnD8='} 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f55b9d0b4ddc15a2-006615c72a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:54:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703274.178131_867.1", 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:54:34.178131Z" 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703274.178131_867.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:34.749+0000 7fe28dc16780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:34.750+0000 7fe28dc16780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703274.178131_867.1", 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:54:34.178131Z", 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:54:34Z" 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.213 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.214 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.215 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1712703274.178131_867.1", 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:54:34.178131Z", 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.217 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:54:34Z" 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.218 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703274.178131_867.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:54:34.178131Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703274.178131_867.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703274.178131_867.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-56 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-56 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-56 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.219 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:35.085+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:40.086+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:45.086+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:50.087+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:54:55.087+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:00.088+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:05.089+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:10.089+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:15.090+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:20.090+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:25.091+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:30.092+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:35.093+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:40.094+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:45.093+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:50.094+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:55:55.094+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:00.095+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:05.095+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:10.096+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:15.096+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:20.097+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:25.097+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:30.098+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:35.099+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:40.099+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:45.100+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:50.100+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:56:55.101+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:00.102+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:05.102+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:10.103+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:15.104+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:20.105+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:25.105+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:30.105+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:35.106+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:40.106+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:45.107+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:50.108+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.224 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi063.stdout: INFO: 2024-04-09T22:57:53.397+0000 7ff8e4c84700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-10T00:01:41.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:57:55.108+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:00.109+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:05.109+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:10.110+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:15.110+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:20.111+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:25.112+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:30.112+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:35.113+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:40.114+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:45.114+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:50.115+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:58:55.116+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:00.117+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:05.117+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:10.117+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:15.118+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:20.118+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:25.119+0000 7fe0fc0e8780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:30.120+0000 7fe0fc0e8780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:30.120+0000 7fe0fc0e8780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:30.120+0000 7fe0fc0e8780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.228 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.229 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.229 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.229 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_incremental 2024-04-10T00:01:41.229 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.230 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.230 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.230 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.230 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1480, in test_bucket_reshard_incremental 2024-04-10T00:01:41.230 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.231 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.231 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.231 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.231 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.231 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.231 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.232 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.232 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/ 2024-04-10T00:01:41.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/ 2024-04-10T00:01:41.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/ 2024-04-10T00:01:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.234 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.234 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.234 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.235 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.235 INFO:tasks.rgw_multisite_tests:/xryqqk-57/ 2024-04-10T00:01:41.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.235 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SM+vEIeJP8I89kbZoWYRGi5quts= 2024-04-10T00:01:41.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SM+vEIeJP8I89kbZoWYRGi5quts=', 'Content-Length': '0'} 2024-04-10T00:01:41.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000007aebc3a5ad464650-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-57 2024-04-10T00:01:41.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703570.166272_914.1", 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T22:59:30.166272Z" 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703570.166272_914.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:30.434+0000 7f071baf7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:30.435+0000 7f071baf7780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703570.166272_914.1", 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T22:59:30.166272Z", 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T22:59:30Z" 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.247 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.248 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "marker": "1_1712703570.166272_914.1", 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T22:59:30.166272Z", 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.250 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.251 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T22:59:30Z" 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703570.166272_914.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T22:59:30.166272Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703570.166272_914.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703570.166272_914.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/ 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/ 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/ 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:/xryqqk-57/ 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ed19f380d11ff09-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/a 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/a 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/a 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.254 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:/xryqqk-57/a 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:dqOnFqwp2XddQJzGEGuBzxGV7h8= 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dqOnFqwp2XddQJzGEGuBzxGV7h8='} 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe6dc4fb378b7a50-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/ 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/ 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/ 2024-04-10T00:01:41.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:/xryqqk-57/ 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-10T00:01:41.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8870d7452f0002d-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/b 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/b 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/b 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.257 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:/xryqqk-57/b 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5HWFV5DZWgq+R802g0ymRCPzNxk= 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5HWFV5DZWgq+R802g0ymRCPzNxk='} 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000410e7199b24fac32-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/ 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/ 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/ 2024-04-10T00:01:41.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:/xryqqk-57/ 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-10T00:01:41.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000555e032fc8d2edd-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/c 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/c 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/c 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.260 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:/xryqqk-57/c 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:opJXVWuK8D8mqAuPaBjSBmJ6hXU= 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:opJXVWuK8D8mqAuPaBjSBmJ6hXU='} 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000069a04d861003af52-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/ 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/ 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/ 2024-04-10T00:01:41.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:/xryqqk-57/ 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.262 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso= 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TXL2fjw7vRRBx4N9nXDbC0tSnso=', 'Content-Length': '0'} 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033ea05df42a7c9a7-006615c852-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-57/d 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-57/d 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-57/d 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.263 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 22:59:30 GMT 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:/xryqqk-57/d 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XkeaYmBVdS9Y8MY/HfpJPldSDvw= 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 22:59:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XkeaYmBVdS9Y8MY/HfpJPldSDvw='} 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000807f10672a21c21e-006615c852-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 22:59:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-57 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.264 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-57 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-57 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:30.797+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:35.798+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:40.798+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:45.799+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.265 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:50.799+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T22:59:55.800+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:00.800+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:05.801+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:10.802+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:15.802+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:20.803+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:25.804+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:30.804+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:35.805+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:40.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.266 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:45.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:50.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:00:55.806+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:00.807+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:05.807+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:10.808+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:15.808+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:20.809+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:25.810+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:30.810+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:35.811+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:40.812+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.267 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:45.812+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:50.813+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:01:55.813+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:00.814+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:05.815+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:10.815+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:15.816+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:20.817+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:25.817+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:30.817+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:35.818+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:40.818+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:45.819+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:50.819+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:02:55.820+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:00.821+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:05.821+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:10.822+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:15.822+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:20.823+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:25.823+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:30.824+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:35.825+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:40.825+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:45.826+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:50.827+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:03:55.827+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:00.828+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:05.829+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:10.829+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:15.829+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:20.830+0000 7f0c36861780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:25.830+0000 7f0c36861780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:25.830+0000 7f0c36861780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:25.830+0000 7f0c36861780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.270 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_full 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.271 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1531, in test_bucket_reshard_full 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:25 GMT 2024-04-10T00:01:41.273 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Y7ny92BpUdFLHdDHMrqeuaNaMdU= 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:25 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Y7ny92BpUdFLHdDHMrqeuaNaMdU=', 'Content-Length': '0'} 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000009884e99599f56db3-006615c979-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-58 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703865.880236_928.1", 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:04:25.880236Z" 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:26.117+0000 7fadaeda8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:26.118+0000 7fadaeda8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.278 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703865.870934_925.1", 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.279 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:04:25.870934Z", 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:04:26Z" 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.281 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.282 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.284 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "marker": "1_1712703865.870934_925.1", 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:04:25.870934Z", 2024-04-10T00:01:41.285 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.286 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.286 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.286 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.286 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.286 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.287 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.287 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.288 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.289 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.289 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:04:26Z" 2024-04-10T00:01:41.289 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.289 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703865.870934_925.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:04:25.870934Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.870934_925.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712703865.880236_928.1 target=1_1712703865.870934_925.1 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:31.252+0000 7fbf48537780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:31.253+0000 7fbf48537780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712703865.880236_928.1", 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:04:25.880236Z", 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.296 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:04:31Z" 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.297 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.298 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "marker": "1_1712703865.880236_928.1", 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:04:25.880236Z", 2024-04-10T00:01:41.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.301 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:04:31Z" 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712703865.880236_928.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:04:25.880236Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712703865.880236_928.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.302 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:04:31.283+0000 7f8b978c8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 49440) UID: 0 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:04:31.283+0000 7f8b9ca9d7c0 -1 shutting down 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003aba5d5d57a69510-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/a 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/a 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/a 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:/xryqqk-58/a 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IWfY3eLOntdWbL5N2GRseQc5vLY= 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IWfY3eLOntdWbL5N2GRseQc5vLY='} 2024-04-10T00:01:41.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056b12ffdbac838f1-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083802fb31b78b52b-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/b 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/b 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/b 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:/xryqqk-58/b 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rszmoV51IqhmBCnlB/M6Ph1aFbs= 2024-04-10T00:01:41.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rszmoV51IqhmBCnlB/M6Ph1aFbs='} 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000450fc04c85f6090c-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.313 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a893907e4152df-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/c 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/c 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/c 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.315 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:/xryqqk-58/c 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vIvcbcP+DRTWRJpNrwytAHNWrqI= 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vIvcbcP+DRTWRJpNrwytAHNWrqI='} 2024-04-10T00:01:41.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af87ea30ca238667-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.318 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000469be5c22fa3ff19-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/d 2024-04-10T00:01:41.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/d 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/d 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:/xryqqk-58/d 2024-04-10T00:01:41.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.322 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4H1z4gVrVaZ6B83iboWhYSWM53U= 2024-04-10T00:01:41.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4H1z4gVrVaZ6B83iboWhYSWM53U='} 2024-04-10T00:01:41.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d0c295c9eb0f884-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'xryqqk-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'xryqqk-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket xryqqk-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-58 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 4 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.324 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000978247dff672cdb3-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/e 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/e 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/e 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:/xryqqk-58/e 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hpfZFfSImcTMXnYq91Pj7wZFMPc= 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hpfZFfSImcTMXnYq91Pj7wZFMPc='} 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001681e049f7510c20-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.327 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000645333ae9380eb17-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/f 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/f 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/f 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:/xryqqk-58/f 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:HnnKczJmRNBZ0qUyqYIdAPKj/d4= 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HnnKczJmRNBZ0qUyqYIdAPKj/d4='} 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000202d12105d460cec-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.330 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031e924043ad36d4b-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/g 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/g 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/g 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:/xryqqk-58/g 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ITA9sGl8xDhbVCnj0+9uWBMg75Q= 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ITA9sGl8xDhbVCnj0+9uWBMg75Q='} 2024-04-10T00:01:41.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009fadb3ad8b2a69a1-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/ 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/ 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/ 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.333 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:/xryqqk-58/ 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk= 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:iyfkICsoX1LmErrjtstdYhul2Fk=', 'Content-Length': '0'} 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c43b615b1c1fe21-006615c985-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-58/h 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-58/h 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-58/h 2024-04-10T00:01:41.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:04:37 GMT 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:/xryqqk-58/h 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.335 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9pvW5QBRcdJ3Lb4tKDRESnFIUWs= 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:04:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9pvW5QBRcdJ3Lb4tKDRESnFIUWs='} 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002bafc4c0127dcf9b-006615c985-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:04:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > curl http://smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > curl http://smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: anonymous 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-58 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-58 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-58 2024-04-10T00:01:41.337 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:39.081+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:44.081+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:49.082+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:54.082+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:04:59.083+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:04.083+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:09.084+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:14.085+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:19.085+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:24.086+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:29.086+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:34.086+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:39.087+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:44.089+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:49.088+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:54.089+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:05:59.090+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:04.090+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:09.091+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:14.091+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:19.092+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:24.092+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:29.093+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:34.094+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:39.094+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:44.095+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:49.095+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:54.096+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:06:59.096+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:04.097+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:09.097+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:14.098+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:19.098+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:24.100+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:29.100+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:34.100+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:39.101+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:44.102+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:49.102+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:54.103+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:07:59.103+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:04.104+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:09.105+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:14.105+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:19.106+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:24.106+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:29.107+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:34.108+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:39.108+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:44.109+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:49.109+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:54.111+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:08:59.111+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:04.112+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:09.113+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:14.113+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:19.113+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:24.114+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:29.114+0000 7f8d3a007780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:34.115+0000 7f8d3a007780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:34.115+0000 7f8d3a007780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:34.115+0000 7f8d3a007780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, then run bucket sync 2024-04-10T00:01:41.343 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1624, in test_bucket_sync_run_basic_incremental 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.344 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:x6jYdnOWEsi5V6O1kEQmgr31AmM= 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x6jYdnOWEsi5V6O1kEQmgr31AmM=', 'Content-Length': '0'} 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ed3284f86aa95ec5-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-59 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704174.442514_998.1", 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:09:34.442514Z" 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704174.442514_998.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:34.660+0000 7ff870192780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:34.661+0000 7ff870192780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.350 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704174.442514_998.1", 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:09:34.442514Z", 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:09:34Z" 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.353 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.354 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.355 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704174.442514_998.1", 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:09:34.442514Z", 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.357 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:09:34Z" 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704174.442514_998.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:09:34.442514Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704174.442514_998.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704174.442514_998.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "objs": [ 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000001", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000005", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000012", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000004", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000011", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000000", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000010", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000003", 2024-04-10T00:01:41.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000013", 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000002", 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "obj00000000000000000006" 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000323abcd6139ccead-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000001 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000001 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000001 2024-04-10T00:01:41.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000001 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.362 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:suTJFYHEYYrHQF20I9S2AvgORdw= 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:suTJFYHEYYrHQF20I9S2AvgORdw='} 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ba891640121f9fd-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.363 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034a8ce7d1a766071-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000012 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000012 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000012 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000012 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xgUcG4goJJLUHkTLLoAailK8a8Y= 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xgUcG4goJJLUHkTLLoAailK8a8Y='} 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ba8bc5997294082-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.366 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000425f8c461467a8e9-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000003 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000003 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000003 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000003 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YM0iWAeyhhLf3Hz1ZfBvn0hkZqc= 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YM0iWAeyhhLf3Hz1ZfBvn0hkZqc='} 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041f8e693eff16b66-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.369 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f206023508fbf67-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000013 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000013 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000013 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000013 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YN2py/4E5deOkX8sTC3CtyLaZus= 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YN2py/4E5deOkX8sTC3CtyLaZus='} 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a4b52410cdd39467-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.372 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc2dfc43ed6485de-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000006 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000006 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000006 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.373 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000006 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:HGURoYFbBbQJ2L3//gZ/HVK9hcI= 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HGURoYFbBbQJ2L3//gZ/HVK9hcI='} 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000881ea156056bfc06-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb4b063d27e465a5-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000000 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000000 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000000 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.376 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000000 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QbwL3huOV9X5VQTtFEoem/u+13I= 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QbwL3huOV9X5VQTtFEoem/u+13I='} 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e70b2a777ff92c2b-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f5fdfbf0135bba1-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000010 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000010 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000010 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000010 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8BLlRr0Aqkzq0LcKFls6NoZn1h0= 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8BLlRr0Aqkzq0LcKFls6NoZn1h0='} 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020133f6a9f04787b-006615caae-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64= 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Asz0ini6eIxCK0O8HN8cTk92Q64=', 'Content-Length': '0'} 2024-04-10T00:01:41.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091497ce79a2af1fc-006615caae-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000005 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000005 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000005 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.382 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:34 GMT 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000005 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mPZrjFMlynhfKiwQ0CY6SLqnTng= 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:34 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mPZrjFMlynhfKiwQ0CY6SLqnTng='} 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093c9e93f0abb0c58-006615caaf-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:35 GMT 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM= 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM=', 'Content-Length': '0'} 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4c3764ec60e6d09-006615caaf-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000004 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000004 2024-04-10T00:01:41.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000004 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.385 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:35 GMT 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000004 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:BTa4LKP+jL6Cpyl+hA+0qt8vqRU= 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BTa4LKP+jL6Cpyl+hA+0qt8vqRU='} 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9f9c4c7b1e857d4-006615caaf-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/ 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/ 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/ 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:35 GMT 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:/xryqqk-59/ 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM= 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gW0LyYoDZCWQ3BNzlcjgMJOY2wM=', 'Content-Length': '0'} 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf1e5aa04729251f-006615caaf-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-59/obj00000000000000000002 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-59/obj00000000000000000002 2024-04-10T00:01:41.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-59/obj00000000000000000002 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:09:35 GMT 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:/xryqqk-59/obj00000000000000000002 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.388 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gGgn44Gmzo/Ywxfn9jnS+gtrrHQ= 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:09:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gGgn44Gmzo/Ywxfn9jnS+gtrrHQ='} 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000419d076e5232b9b1-006615caaf-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:09:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-59 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-59 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-59 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:35.321+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:40.322+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:45.323+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:50.323+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:09:55.323+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:00.324+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:05.325+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:10.325+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:15.325+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:20.326+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:25.327+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:30.327+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:35.327+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:40.328+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:45.329+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:50.330+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:10:55.329+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:00.330+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:05.331+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:10.331+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:15.332+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:20.333+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:25.333+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:30.334+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:35.334+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:40.335+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:45.335+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:50.336+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:11:55.336+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:00.337+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:05.338+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:10.338+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:15.339+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:20.340+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:25.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:30.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:35.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:40.341+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:45.342+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:50.342+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:12:55.343+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:00.343+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:05.344+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:10.345+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:15.345+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:20.346+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:25.346+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:30.347+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:35.348+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:40.348+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:45.349+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:50.350+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:13:55.351+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:00.351+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:05.352+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:10.353+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:15.352+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:20.353+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:25.353+0000 7f12f4aeb780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:30.354+0000 7f12f4aeb780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:30.354+0000 7f12f4aeb780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:30.354+0000 7f12f4aeb780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.394 INFO:tasks.rgw_multisite_tests:FAIL: Create several generations of objects, trash them, then run bucket sync init 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1691, in test_zap_init_bucket_sync_run 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.395 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/ 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/ 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/ 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:/xryqqk-60/ 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lq/dufZIAuvmM0IKNkfvb1XflE4= 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lq/dufZIAuvmM0IKNkfvb1XflE4=', 'Content-Length': '0'} 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000f87ef45d4bc24c8e-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-60 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:42:43.140128Z" 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "0.000000" 2024-04-10T00:01:41.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704470.400395_1051.1", 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:14:30.400395Z" 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.399 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:30.639+0000 7fc2810b6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:30.640+0000 7fc2810b6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "", 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "0.000000", 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704470.400395_1051.1", 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:14:30.400395Z", 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:14:30Z" 2024-04-10T00:01:41.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.407 INFO:tasks.rgw_multisite_tests: "marker": "1_1712698963.140128_76.1", 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:42:43.140128Z", 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.408 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704470.400395_1051.1", 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.409 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:14:30.400395Z", 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.410 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:14:30Z" 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712698963.140128_76.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T21:42:43.140128Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704470.400395_1051.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:14:30.400395Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.411 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712698963.140128_76.1', 1: '', 2: '1_1712704470.400395_1051.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/ 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/ 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/ 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.412 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:/xryqqk-60/ 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000168fd2391e264844-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/obj11 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/obj11 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/obj11 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:/xryqqk-60/obj11 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nC5XBjwtnsIwH7LyWBzHu9r/PVo= 2024-04-10T00:01:41.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nC5XBjwtnsIwH7LyWBzHu9r/PVo='} 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff2d42046be13aae-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/ 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/ 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/ 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.415 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:/xryqqk-60/ 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3021814b0f027d1-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/obj22 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/obj22 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/obj22 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:/xryqqk-60/obj22 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:9PrSV00GXEJre3CPfNgFAKfwL24= 2024-04-10T00:01:41.417 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:9PrSV00GXEJre3CPfNgFAKfwL24='} 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026823a0ac558d154-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/ 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/ 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/ 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.418 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:/xryqqk-60/ 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000532894accdd0dce7-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/obj33 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/obj33 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/obj33 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.419 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:/xryqqk-60/obj33 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OIqTdgeskTznUhrKwTqRT951Grc= 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OIqTdgeskTznUhrKwTqRT951Grc='} 2024-04-10T00:01:41.420 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009edefdba884e778f-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/ 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/ 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/ 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.421 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:/xryqqk-60/ 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000461b41f436a23b09-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/obj44 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/obj44 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/obj44 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.422 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:/xryqqk-60/obj44 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XVnAZP+DxjGJYAVWSqQUBUS0Ano= 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XVnAZP+DxjGJYAVWSqQUBUS0Ano='} 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056529d813a7c2b13-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/ 2024-04-10T00:01:41.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/ 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/ 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.424 INFO:tasks.rgw_multisite_tests:/xryqqk-60/ 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw= 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d+tEcD2yPBclJ90Wdic26z9vxVw=', 'Content-Length': '0'} 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000024272e217fef2290-006615cbd6-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-60/obj55 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-60/obj55 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-60/obj55 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:14:30 GMT 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:/xryqqk-60/obj55 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:x4qnx2TNCfXWs9fDyI3rDJVwfFg= 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:14:30 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x4qnx2TNCfXWs9fDyI3rDJVwfFg='} 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000430cdf65f80b10db-006615cbd6-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:14:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-60 2024-04-10T00:01:41.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-60 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-60 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:30.988+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:35.989+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:40.989+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:45.990+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:50.990+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:14:55.991+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:00.992+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:05.992+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:10.993+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:15.993+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:20.994+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:25.995+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:30.995+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:35.996+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:40.997+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:45.998+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:50.997+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:15:55.998+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:00.998+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:05.999+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:10.999+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:16.000+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:21.000+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:26.001+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:31.001+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:36.002+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:41.003+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:46.003+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:51.004+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:16:56.004+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:01.005+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:06.006+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:11.006+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:16.007+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:21.008+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:26.009+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:31.008+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:36.009+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:41.009+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:46.010+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:51.011+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:17:56.011+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:01.012+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:06.013+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:11.013+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:16.014+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:21.014+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:26.015+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:31.016+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:36.016+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:41.017+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:46.017+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:51.018+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:18:56.018+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:01.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:06.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:11.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:16.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:21.020+0000 7f407fb69780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:26.021+0000 7f407fb69780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:26.021+0000 7f407fb69780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:26.021+0000 7f407fb69780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.432 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_full_sync_after_data_sync_init 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1769, in test_bucket_full_sync_after_data_sync_init 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.433 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/ 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/ 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/ 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.434 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.435 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:26 GMT 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:/xryqqk-61/ 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3urQGhh6NcnIzSRU3gLwjFHC8x4= 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:26 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3urQGhh6NcnIzSRU3gLwjFHC8x4=', 'Content-Length': '0'} 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006f595e70eb2ada1f-006615ccfe-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-61 2024-04-10T00:01:41.436 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.368163_1115.1", 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.368163Z" 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704766.368163_1115.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:26.578+0000 7f3000cbe780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:26.579+0000 7f3000cbe780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.368163_1115.1", 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.368163Z", 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:19:26Z" 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.444 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.445 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.368163_1115.1", 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.368163Z", 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.447 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.448 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:19:26Z" 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712704766.368163_1115.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.368163Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704766.368163_1115.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712704766.368163_1115.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:19:26.609+0000 7f4866209700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 101072) UID: 0 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:19:26.609+0000 7f486b3de7c0 -1 shutting down 2024-04-10T00:01:41.449 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/ 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/ 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/ 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:/xryqqk-61/ 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000596a7ab655943e91-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/obj11 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/obj11 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/obj11 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:/xryqqk-61/obj11 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.452 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TqPEeIMRTGpqIKzWrd/6G9DP4Mc= 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TqPEeIMRTGpqIKzWrd/6G9DP4Mc='} 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054f1333d7af2a600-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/ 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/ 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/ 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.453 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:/xryqqk-61/ 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e3f6609af8d0ac1-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/obj22 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/obj22 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/obj22 2024-04-10T00:01:41.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.455 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:/xryqqk-61/obj22 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2bftUdGYQ9hgTQc51O92gD+mZBc= 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2bftUdGYQ9hgTQc51O92gD+mZBc='} 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f419afeab475e773-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/ 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/ 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/ 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:/xryqqk-61/ 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092be558bd0cdc088-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/obj33 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/obj33 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/obj33 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.458 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:/xryqqk-61/obj33 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5d1XpjqbXVS5nwlI6BLF2ibObwo= 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5d1XpjqbXVS5nwlI6BLF2ibObwo='} 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001b2298938c13f29e-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/ 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/ 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/ 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:/xryqqk-61/ 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005acccc590bc890f-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/obj44 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/obj44 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/obj44 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.461 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:/xryqqk-61/obj44 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:O965TUr4uWgje6omCB7u9isY1iE= 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:O965TUr4uWgje6omCB7u9isY1iE='} 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ae3259341e51d6e-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/ 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/ 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/ 2024-04-10T00:01:41.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.463 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:/xryqqk-61/ 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys= 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rf65J2mHJM1FpFURTqz4U2Ds+ys=', 'Content-Length': '0'} 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0dc3796901d7bba-006615cd04-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-61/obj55 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-61/obj55 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-61/obj55 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.466 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.466 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.466 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.467 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:19:32 GMT 2024-04-10T00:01:41.467 INFO:tasks.rgw_multisite_tests:/xryqqk-61/obj55 2024-04-10T00:01:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.467 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gMdB0DMgROcSkMT7H/htiZ/xCWU= 2024-04-10T00:01:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:19:32 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gMdB0DMgROcSkMT7H/htiZ/xCWU='} 2024-04-10T00:01:41.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa81b19ac7637ae5-006615cd04-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:19:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.468 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.468 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:32.967+0000 7fdb09d6f780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:32.969+0000 7fdb09d6f780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > curl http://smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.469 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-10T00:01:41.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-10T00:01:41.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-10T00:01:41.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-10T00:01:41.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > curl http://smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.470 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-10T00:01:41.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-10T00:01:41.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 14266 0 --:--:-- --:--:-- --:--:-- 14266 2024-04-10T00:01:41.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: anonymous 2024-04-10T00:01:41.471 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-61 2024-04-10T00:01:41.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.471 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-61 2024-04-10T00:01:41.472 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-61 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:34.329+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:39.330+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:44.330+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:49.331+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:54.331+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:19:59.331+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:04.332+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:09.333+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:14.333+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:19.334+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:24.334+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:29.335+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:34.336+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:39.337+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:44.338+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:49.337+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:54.338+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:20:59.339+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:04.339+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:09.340+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:14.340+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:19.341+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:24.341+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:29.342+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:34.342+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:39.343+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:44.343+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:49.344+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:54.345+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:21:59.345+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:04.346+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:09.346+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:14.347+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:19.348+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:24.348+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:29.348+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:34.349+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:39.349+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:44.350+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:49.350+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:54.351+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:22:59.352+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:04.352+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:09.353+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:14.353+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:19.354+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:24.355+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:29.355+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:34.356+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:39.356+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:44.357+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:49.357+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:54.358+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:23:59.359+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:04.359+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:09.360+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:14.360+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:19.361+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:24.361+0000 7f3d18315780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:29.362+0000 7f3d18315780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:29.362+0000 7f3d18315780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:29.362+0000 7f3d18315780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.480 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.481 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.481 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.481 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init 2024-04-10T00:01:41.481 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.481 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1809, in test_resharded_bucket_full_sync_after_data_sync_init 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.483 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:29 GMT 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uz3ibZJRqzxQj7qgrh0vzMBjsxA= 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:29 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uz3ibZJRqzxQj7qgrh0vzMBjsxA=', 'Content-Length': '0'} 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000954e640745db255c-006615ce2d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-62 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705069.409525_1169.1", 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:24:29.409525Z" 2024-04-10T00:01:41.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:29.645+0000 7fc454565780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:29.646+0000 7fc454565780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705069.400619_1166.1", 2024-04-10T00:01:41.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:24:29.400619Z", 2024-04-10T00:01:41.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:24:29Z" 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.497 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.498 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1712705069.400619_1166.1", 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:24:29.400619Z", 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:24:29Z" 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705069.400619_1166.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:24:29.400619Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.400619_1166.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712705069.409525_1169.1 target=1_1712705069.400619_1166.1 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:34.793+0000 7f3f835f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:34.795+0000 7f3f835f5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705069.409525_1169.1", 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:24:29.409525Z", 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:24:34Z" 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.508 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "marker": "1_1712705069.409525_1169.1", 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:24:29.409525Z", 2024-04-10T00:01:41.511 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.512 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:24:34Z" 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705069.409525_1169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:24:29.409525Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705069.409525_1169.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.513 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:24:34.820+0000 7f2561ab8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 103748) UID: 0 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:24:34.820+0000 7f2566c8d7c0 -1 shutting down 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-10T00:01:41.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cd14c5ace93f8b2-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj11 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj11 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj11 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.516 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj11 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:s2R5E5C//4lsLIXQ68IY20gMl3g= 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:s2R5E5C//4lsLIXQ68IY20gMl3g='} 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082d8f42157a5c64b-006615ce38-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-10T00:01:41.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7e90f816c51905c-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj22 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj22 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj22 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.519 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj22 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:O4GiUvFc8ezkWe8+11qbt+y8Xbo= 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:O4GiUvFc8ezkWe8+11qbt+y8Xbo='} 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7962aaac9415477-006615ce38-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83212c03e156bce-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj33 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj33 2024-04-10T00:01:41.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj33 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.522 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.523 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.523 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.523 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.523 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj33 2024-04-10T00:01:41.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.523 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uD3/bDzRANsk2RHW+fP52Fltoqc= 2024-04-10T00:01:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uD3/bDzRANsk2RHW+fP52Fltoqc='} 2024-04-10T00:01:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d6f1f9fd74e268be-006615ce38-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.526 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.526 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.526 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.526 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8= 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gvniHslQqvd2lIe5fvM57v2W2/8=', 'Content-Length': '0'} 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068f761486e4bec5e-006615ce38-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj44 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj44 2024-04-10T00:01:41.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj44 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:40 GMT 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj44 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hXOkIVCfE8wtwE45MRJROQlEFE0= 2024-04-10T00:01:41.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:40 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hXOkIVCfE8wtwE45MRJROQlEFE0='} 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cde588f179708dfe-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.531 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001fcb5ef84262ff98-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj55 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj55 2024-04-10T00:01:41.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj55 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj55 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:77LTd+pv0NNv6pRvjO25OtrXijU= 2024-04-10T00:01:41.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:77LTd+pv0NNv6pRvjO25OtrXijU='} 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043b1a8dc7ac4f145-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-62 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 5 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.537 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067d9e03502b7a116-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj17 2024-04-10T00:01:41.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj17 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj17 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj17 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.540 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:n4O5SGyz69/eKEbkpNpP/HEiqzg= 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:n4O5SGyz69/eKEbkpNpP/HEiqzg='} 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e49e1a04fd07d115-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.541 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9088f5dbdda0ab1-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj34 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj34 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj34 2024-04-10T00:01:41.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.543 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj34 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hrdzG4U3NrheRN9Hwen/AuZ32t4= 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hrdzG4U3NrheRN9Hwen/AuZ32t4='} 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b220d1e24907eed-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd5d5ce7838b2770-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj51 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj51 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj51 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj51 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/s7hAi9/tZFxSfIWaS/Rt3kw6no= 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/s7hAi9/tZFxSfIWaS/Rt3kw6no='} 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000848cc47cd93ed4fd-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001185004c1f994d96-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj68 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj68 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj68 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.549 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj68 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2zJ2Rfa16hJ/lnVTL5cduC4e6Dc= 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2zJ2Rfa16hJ/lnVTL5cduC4e6Dc='} 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063f7d4503749fde8-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000713a63e99c031a61-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj85 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj85 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj85 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj85 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:WIJQQb7G++3Lq04Qe4Vqhl6ZpQE= 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:WIJQQb7G++3Lq04Qe4Vqhl6ZpQE='} 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000720acebcfeec36da-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-62 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 10 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7d2bece1a91ef00-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj19 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj19 2024-04-10T00:01:41.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj19 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.556 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj19 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r5vxXGaUYx8HTpAnPizIDh3FZo4= 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r5vxXGaUYx8HTpAnPizIDh3FZo4='} 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fd2e4b31dff5f46-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f038931c63a5e447-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj38 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj38 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj38 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj38 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.559 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cl6TJypExdqdghWxguro/5ZfeOk= 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cl6TJypExdqdghWxguro/5ZfeOk='} 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009194b2952c603ef0-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.560 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c033fdeb29f9fc91-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj57 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj57 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj57 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj57 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.562 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:7A+wfCpyDoAvc1O+2SHB9v876EA= 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:7A+wfCpyDoAvc1O+2SHB9v876EA='} 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000551e6235a70b21de-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.563 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042c940f0ca69221e-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj76 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj76 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj76 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj76 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/WycllqdqPbwyGJlvEUCegAlgWs= 2024-04-10T00:01:41.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/WycllqdqPbwyGJlvEUCegAlgWs='} 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008da02ddd67c7f757-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092325ffc09b01458-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj95 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj95 2024-04-10T00:01:41.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj95 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.568 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj95 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:4jb8naZFiSkag03GHfanb9x9dco= 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:4jb8naZFiSkag03GHfanb9x9dco='} 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1698b54ead1dacd-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-62 2024-04-10T00:01:41.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 15 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033bbbc7f85ba73b7-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj23 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj23 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj23 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj23 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QK3DsefEOZ+gMKYvSiJaoweaXJM= 2024-04-10T00:01:41.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QK3DsefEOZ+gMKYvSiJaoweaXJM='} 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038232fab3b9146f4-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006843fa4decbdf7bd-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj46 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj46 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj46 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj46 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bbSJNNDQL9w6NbKr8N1SLVoFUTQ= 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bbSJNNDQL9w6NbKr8N1SLVoFUTQ='} 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004e11e579b99d272-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.577 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q= 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:8LykTep7afXPatMqLZ9yvL0bv1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8adf1e3051272f3-006615ce39-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj69 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj69 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj69 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:41 GMT 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj69 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qBzLxAkDk7WlZXrexzLfjfgsbkc= 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:41 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qBzLxAkDk7WlZXrexzLfjfgsbkc='} 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000414119d22d09363b-006615ce39-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2a34559cfb23081-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj92 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj92 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj92 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj92 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:dV6V9xvMT8dgG18tmFeZvTPwzTQ= 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dV6V9xvMT8dgG18tmFeZvTPwzTQ='} 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091a9d8bc05033b4c-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019f9228bfeb1fcc2-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj115 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj115 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj115 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj115 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:x/KA0lAxxC/Y+sVELQYRPf/aaV0= 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x/KA0lAxxC/Y+sVELQYRPf/aaV0='} 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c887604b3d8225e-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-62 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 20 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5c2f0abcb845d59-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj29 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj29 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj29 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.588 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj29 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:r/JeDXNTypn6Vy67TZ8e5Ju1jiM= 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:r/JeDXNTypn6Vy67TZ8e5Ju1jiM='} 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e0ac59af23c32bd-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e95cb797e6e6bd4-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj58 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj58 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj58 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.591 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj58 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EyoLJHbQqPoJoiAWlq0qUR2PISo= 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EyoLJHbQqPoJoiAWlq0qUR2PISo='} 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea0edede445bc863-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.593 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6ff1cda1185419f-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj87 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj87 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj87 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj87 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ki0zpER0Nsq76EOZ1jtiZlH5ZNQ= 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ki0zpER0Nsq76EOZ1jtiZlH5ZNQ='} 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082d910e2e6651aba-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.596 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000124563e84a2f2f19-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj116 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj116 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj116 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj116 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:reoom6f+AY+tXslaWxj9PSfyJU8= 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:reoom6f+AY+tXslaWxj9PSfyJU8='} 2024-04-10T00:01:41.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ddbad886b4d52d2-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.599 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d52caa1b79f867f-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj145 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj145 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj145 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj145 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:T+dIcypfxJNiKV9uXFGYFK0u/Ms= 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:T+dIcypfxJNiKV9uXFGYFK0u/Ms='} 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b760b021fc17191d-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-62 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 25 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017c6f03e2c71c79b-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj31 2024-04-10T00:01:41.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj31 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj31 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj31 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:HbxIG8Y7PB2LydBX4xCp7tM/a70= 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:HbxIG8Y7PB2LydBX4xCp7tM/a70='} 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b687462dfc2b891-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.606 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcf61b548f5a155-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj62 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj62 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj62 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.609 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.609 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.609 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj62 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TpsfjG146XAC4Tm49nPRVIlYxdw= 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TpsfjG146XAC4Tm49nPRVIlYxdw='} 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001d94f128c47610d8-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae0e1e6109a32ac3-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj93 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj93 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj93 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.612 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj93 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:sirb4clRUNlQLg4HQ1poRmkV8UQ= 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:sirb4clRUNlQLg4HQ1poRmkV8UQ='} 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056d218329b1e35bf-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8e38ae761de6ed9-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj124 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj124 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj124 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj124 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qAzgQq2TKZNcllHHqoExcPQ4tb0= 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qAzgQq2TKZNcllHHqoExcPQ4tb0='} 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c73ba4ca1c945d3d-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.617 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q= 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:TxOwKMj+fZATqLsaHfHeomsIY3Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8c601c45c272d15-006615ce3a-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj155 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj155 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj155 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:42 GMT 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj155 2024-04-10T00:01:41.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:EnEOVB8H6DWIcHZwZbdmVyXsS0o= 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:42 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:EnEOVB8H6DWIcHZwZbdmVyXsS0o='} 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f20d768cdb960ee-006615ce3a-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-62', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-62 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-62 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 30 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.621 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a87fc49616a3f43-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj37 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj37 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj37 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj37 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SlvWFtEF01uE+ftpbeENub0eNdQ= 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SlvWFtEF01uE+ftpbeENub0eNdQ='} 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a9debe9216021f4-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.624 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008417dece65ec2494-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj74 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj74 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj74 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj74 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:db7Pm8ncuVP/p/h+URq4Mkxev6o= 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:db7Pm8ncuVP/p/h+URq4Mkxev6o='} 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000386233700cb5eb81-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.627 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000692b7c9927560358-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj111 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj111 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj111 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj111 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:dC30iN+CdgoZ2f+2cnRRIb0L30k= 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:dC30iN+CdgoZ2f+2cnRRIb0L30k='} 2024-04-10T00:01:41.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd5c7aaf8f875ce5-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.630 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000015fb92bc16cef68a-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj148 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj148 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj148 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj148 2024-04-10T00:01:41.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:holrHiDaYjTebevyoAZfABRSNP8= 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:holrHiDaYjTebevyoAZfABRSNP8='} 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bccbb30ccf84379a-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/ 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/ 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/ 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:/xryqqk-62/ 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q= 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:i743BmRIP6AdnJAcTUlF0LQBQ1Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6efc7fb54a8ee28-006615ce3b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-62/obj185 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-62/obj185 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-62/obj185 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.635 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:24:43 GMT 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:/xryqqk-62/obj185 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:UpgfzssyV53lcI26dGozEc6Xv3U= 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:24:43 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:UpgfzssyV53lcI26dGozEc6Xv3U='} 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081441446340ffe5e-006615ce3b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:24:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:43.279+0000 7ff672c7f780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:43.280+0000 7ff672c7f780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > curl http://smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > curl http://smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-10T00:01:41.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: anonymous 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-62 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-62 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-62 2024-04-10T00:01:41.638 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:44.621+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:49.621+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:54.622+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:24:59.622+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:04.623+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:09.624+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:14.624+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:19.625+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:24.625+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:29.626+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:34.627+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:39.628+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:44.627+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:49.628+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:54.628+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:25:59.629+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:04.630+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:09.630+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:14.631+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:19.632+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:24.632+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:29.633+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:34.634+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:39.634+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:44.635+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:49.635+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:54.635+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:26:59.636+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:04.637+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:09.637+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:14.638+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:19.639+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:24.639+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:29.639+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:34.640+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:39.641+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:44.641+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:49.642+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:54.642+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:27:59.643+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:04.644+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:09.645+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:14.645+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:19.646+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:24.647+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:29.647+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:34.647+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:39.648+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:44.649+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:49.650+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:54.651+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:28:59.650+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:04.651+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:09.652+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:14.652+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:19.653+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:24.653+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:29.654+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:34.654+0000 7f5019b36780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:39.655+0000 7f5019b36780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:39.655+0000 7f5019b36780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:39.655+0000 7f5019b36780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.644 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1827, in test_bucket_incremental_sync_after_data_sync_init 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.645 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-63/ 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-63/ 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-63/ 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:29:39 GMT 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:/xryqqk-63/ 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Z3XOUPghnUH5qXx89WD/RPZwC3A= 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:29:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Z3XOUPghnUH5qXx89WD/RPZwC3A=', 'Content-Length': '0'} 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000002e13d850796e6b54-006615cf63-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:29:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-63 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705379.699621_1231.1", 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:29:39.699621Z" 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705379.699621_1231.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:39.933+0000 7f869af6d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:39.934+0000 7f869af6d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705379.699621_1231.1", 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:29:39.699621Z", 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:29:39Z" 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.657 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.658 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.659 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "marker": "1_1712705379.699621_1231.1", 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:29:39.699621Z", 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.660 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.661 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:29:39Z" 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705379.699621_1231.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:29:39.699621Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705379.699621_1231.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705379.699621_1231.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-63/ 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-63/ 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-63/ 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:29:39 GMT 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:/xryqqk-63/ 2024-04-10T00:01:41.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DYU0O92Aji0V9r+Gnuc29lT4Ndk= 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:29:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DYU0O92Aji0V9r+Gnuc29lT4Ndk=', 'Content-Length': '0'} 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a634cc968ff6dead-006615cf63-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:29:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-63/dummy 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-63/dummy 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-63/dummy 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:29:39 GMT 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:/xryqqk-63/dummy 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:6GXtccg/BdFSnkczDc0x/oitOSk= 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:29:39 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:6GXtccg/BdFSnkczDc0x/oitOSk='} 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000511b5e87734a0375-006615cf63-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:29:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-63 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.665 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-63 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-63 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:40.188+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:45.188+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.666 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:50.189+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:29:55.190+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:00.190+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:05.191+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:10.192+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:15.192+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:20.193+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:25.193+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:30.194+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:35.195+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:40.194+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.667 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:45.195+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:50.196+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:30:55.196+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:00.197+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:05.197+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:10.198+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:15.198+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:20.199+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:25.199+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:30.200+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:35.200+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.668 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:40.201+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:45.202+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:50.202+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:31:55.203+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:00.203+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:05.205+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:10.205+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:15.204+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:20.205+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:25.206+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:30.206+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:35.207+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.669 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:40.207+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:45.208+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:50.208+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:32:55.209+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:00.209+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:05.210+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:10.211+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:15.211+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:20.212+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:25.213+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:30.214+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.670 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:35.214+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:40.215+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:45.216+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:50.216+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:33:55.216+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:00.217+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:05.217+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:10.218+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:15.218+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:20.219+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:25.219+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.671 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:30.220+0000 7f0f89895780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:35.220+0000 7f0f89895780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:35.220+0000 7f0f89895780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:35.220+0000 7f0f89895780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-10T00:01:41.672 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.673 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tw63IuQ6HHRQv8MJGvQPeB7f1zQ= 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tw63IuQ6HHRQv8MJGvQPeB7f1zQ=', 'Content-Length': '0'} 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000014d252e525377c33-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-64 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.675 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705675.283043_1240.1", 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:34:35.283043Z" 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705675.283043_1240.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.677 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:35.590+0000 7fdfde81d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:35.591+0000 7fdfde81d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.680 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705675.283043_1240.1", 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:35.283043Z", 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.681 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.682 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:34:35Z" 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.683 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.684 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.685 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "marker": "1_1712705675.283043_1240.1", 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:35.283043Z", 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.686 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.687 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:34:35Z" 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705675.283043_1240.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:34:35.283043Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705675.283043_1240.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705675.283043_1240.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.688 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.689 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1ed2334079330d9-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj11 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj11 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj11 2024-04-10T00:01:41.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj11 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.691 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IDF6NWO/r0J4eHFSZwbpHqipkfU= 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IDF6NWO/r0J4eHFSZwbpHqipkfU='} 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c0356396e170990-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3fd00b37e80d432-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj22 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj22 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj22 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.694 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj22 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tdgQ5HpeStYowPptTl/1qbG5SAE= 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tdgQ5HpeStYowPptTl/1qbG5SAE='} 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000067e05189d092a109-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084c5f6fd2e9f2fdd-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj33 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj33 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj33 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj33 2024-04-10T00:01:41.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:a+rT6a916sLuj1QNad+HIDNN03o= 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:a+rT6a916sLuj1QNad+HIDNN03o='} 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000295a35bb9d932869-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d129e3f865c7765e-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj44 2024-04-10T00:01:41.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj44 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj44 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.700 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj44 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:k79K67CQlXcvC60iXFU4WO42wRI= 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:k79K67CQlXcvC60iXFU4WO42wRI='} 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bfb615b997c44e3-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000739c58cceca45116-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj55 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj55 2024-04-10T00:01:41.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj55 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj55 2024-04-10T00:01:41.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:chYURlqhjraLWrGmPNOZbJQIypY= 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:chYURlqhjraLWrGmPNOZbJQIypY='} 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e9ee2c3a7f839b7-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-64 2024-04-10T00:01:41.704 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 5 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.705 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095bff9d38657d6c8-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj17 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj17 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj17 2024-04-10T00:01:41.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj17 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ziig3LRYwUO0wMiDk1AA5EEeDLE= 2024-04-10T00:01:41.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ziig3LRYwUO0wMiDk1AA5EEeDLE='} 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a5705aa950e166bc-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001383461bd0d2edc2-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj34 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj34 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj34 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.710 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj34 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nCPcF899NL1dHcs/Wrzh4sfSrqY= 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nCPcF899NL1dHcs/Wrzh4sfSrqY='} 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003536fc4184fb2950-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070ac447d38d97ef2-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj51 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj51 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj51 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.713 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj51 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1zDB7ToT0ZqdfAFPnXhs8edGBf4= 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1zDB7ToT0ZqdfAFPnXhs8edGBf4='} 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9837026c264d9b1-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0= 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:gqQZsKcwUG9LVLhhPtvMWhbnUL0=', 'Content-Length': '0'} 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b7d0a83ae885d11-006615d08b-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj68 2024-04-10T00:01:41.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj68 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj68 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.716 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:35 GMT 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj68 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:PpA9eB/3nydt93EKfo+/4UnyRBs= 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:35 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:PpA9eB/3nydt93EKfo+/4UnyRBs='} 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f03931dfb51191a5-006615d08b-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.718 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4d7a73640ea5201-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj85 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj85 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj85 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj85 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Z76g5zGzd6q2TUppdfZxBakEEg8= 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Z76g5zGzd6q2TUppdfZxBakEEg8='} 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054836dd1af3191fd-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-64 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 10 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5cca92d07be59cc-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj19 2024-04-10T00:01:41.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj19 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj19 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.723 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj19 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:YnC9zango27PPT2+3CPCORs2pmo= 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:YnC9zango27PPT2+3CPCORs2pmo='} 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a170b77f5886728-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000305bbb07eb39dd36-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj38 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj38 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj38 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj38 2024-04-10T00:01:41.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:2lakgoHynB2dKdwpivhlDhh3PO0= 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:2lakgoHynB2dKdwpivhlDhh3PO0='} 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d45b48e618b42ecb-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ea95934059a07c7-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj57 2024-04-10T00:01:41.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj57 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj57 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.729 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj57 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xZbjDFvCPvr3ebraKS/ggAoE+N0= 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xZbjDFvCPvr3ebraKS/ggAoE+N0='} 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000defaf95de22aa461-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dd7f4974297b18f-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj76 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj76 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj76 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.732 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj76 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:pMc3lO/Evzdvb95uctGRlcjBKPg= 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:pMc3lO/Evzdvb95uctGRlcjBKPg='} 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079b1af0b99f1828a-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.734 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e83a1280d844dafb-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj95 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj95 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj95 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.735 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj95 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:hfwP9/EYmNFhmbRzGMXN+lMj63M= 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:hfwP9/EYmNFhmbRzGMXN+lMj63M='} 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8ce039c5769f215-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-64 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 15 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.737 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001159cc04f75bdbe4-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj23 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj23 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj23 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj23 2024-04-10T00:01:41.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:u2e1iwTtA25czYHn1iUTOdwEA+E= 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:u2e1iwTtA25czYHn1iUTOdwEA+E='} 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b68577430e9e8ac6-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039670f7b54b76a60-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj46 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj46 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj46 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.742 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj46 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/ojwIYoaE/QX/w0WZmXBo9VqTig= 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/ojwIYoaE/QX/w0WZmXBo9VqTig='} 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000776094936df42fd4-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005be1cab099fe8f02-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj69 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj69 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj69 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.745 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj69 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SYrNP2sIsBLI8HAX/5I/PrxhUMc= 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SYrNP2sIsBLI8HAX/5I/PrxhUMc='} 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2affa1733dbdab7-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001df36eab3a002072-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj92 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj92 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj92 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.748 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj92 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:x1zMmTVxt1tkixHdAJ5JP0Hr/pw= 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:x1zMmTVxt1tkixHdAJ5JP0Hr/pw='} 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092bf78b7e6216f41-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.750 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004df3d97750c65999-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj115 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj115 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj115 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj115 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:RmSmU68XCYMK3LoYsUopW+KAyTg= 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:RmSmU68XCYMK3LoYsUopW+KAyTg='} 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee8359bda8e7b010-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.752 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-64 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 20 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa2b801e9326770f-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj29 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj29 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj29 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.755 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj29 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:juepciHOyuiEYlYu655PMOtksvk= 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:juepciHOyuiEYlYu655PMOtksvk='} 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000050734d95420946d5-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI= 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:nwC2DtER3xIGbr9CcTxjFo1kzNI=', 'Content-Length': '0'} 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066d007adac11b25a-006615d08c-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj58 2024-04-10T00:01:41.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj58 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj58 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:36 GMT 2024-04-10T00:01:41.758 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj58 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qWpjETrvg+wa6hkCq5Cj0oYE6L4= 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:36 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qWpjETrvg+wa6hkCq5Cj0oYE6L4='} 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a8d851bb805d4d14-006615d08c-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba3020747287cb58-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj87 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj87 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj87 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.761 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj87 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:J3E3P+rBCWhbY9oqU+S5Hb1rL6I= 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:J3E3P+rBCWhbY9oqU+S5Hb1rL6I='} 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6c26aaed0ce32d5-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f57595421f61b8fd-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj116 2024-04-10T00:01:41.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj116 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj116 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.764 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj116 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:mRsVH0p++4dZbopMUeXvCV6z0QE= 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:mRsVH0p++4dZbopMUeXvCV6z0QE='} 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001942a65b68654482-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.767 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.768 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000556850af91727f6e-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj145 2024-04-10T00:01:41.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj145 2024-04-10T00:01:41.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj145 2024-04-10T00:01:41.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.770 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.770 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.770 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj145 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rjC+CqK0PMgoJ9s3zncg9IBQ3io= 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rjC+CqK0PMgoJ9s3zncg9IBQ3io='} 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dab8de6c534541b4-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.771 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-64 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 25 2024-04-10T00:01:41.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.774 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.774 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.775 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.775 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.775 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.775 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cca077a743a9a8f-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj31 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj31 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj31 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj31 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rpEbOPi/T2U2f/ipgOFpGCcwHJo= 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rpEbOPi/T2U2f/ipgOFpGCcwHJo='} 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7122bdbf7b5aeb0-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.778 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008eece30e08ac2a31-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj62 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj62 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj62 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj62 2024-04-10T00:01:41.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:z5aoDK3CMtp+vZ49HaRCbYSzd2Y= 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:z5aoDK3CMtp+vZ49HaRCbYSzd2Y='} 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001ff8f405a2d6c32-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a39db7690075fe4-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj93 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj93 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj93 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.783 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj93 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:atFlCq+jB6FzDtW+fT6xjE0wx1Y= 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:atFlCq+jB6FzDtW+fT6xjE0wx1Y='} 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057bc48e535bbe36e-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.785 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d885ffc928bff24-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj124 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj124 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj124 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj124 2024-04-10T00:01:41.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3MosNn5w7k0SpykX94ARWepPzoQ= 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3MosNn5w7k0SpykX94ARWepPzoQ='} 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e29d28d5ed56c8cb-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.789 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9f4086980dc2d1b-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj155 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj155 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj155 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj155 2024-04-10T00:01:41.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:XlUTGErfodhlvEHObyyh+ZEIrMo= 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:XlUTGErfodhlvEHObyyh+ZEIrMo='} 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b3d2b00ecb0b159-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'xryqqk-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket xryqqk-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: tenant: 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: bucket name: xryqqk-64 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: total entries: 30 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.794 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.794 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.794 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.794 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e498c88a42cc5281-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj37 2024-04-10T00:01:41.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj37 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj37 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj37 2024-04-10T00:01:41.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:b/7WV2812o39jBioYbdD7YvR89c= 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:b/7WV2812o39jBioYbdD7YvR89c='} 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000120ec679c7d62170-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ff57aaa1abb79b0-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj74 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj74 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj74 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj74 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.799 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:GRngGtyVS4HdzcXzYadezOje2SI= 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:GRngGtyVS4HdzcXzYadezOje2SI='} 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026ec3c6b2334e326-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.800 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9786b171b9e74ab-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj111 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj111 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj111 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj111 2024-04-10T00:01:41.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:f/cEKaRlxFEcmeHMWVJg/MRqW8o= 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:f/cEKaRlxFEcmeHMWVJg/MRqW8o='} 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe06410da11ccebf-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000363fff476e221c1c-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj148 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj148 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj148 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj148 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.805 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:zysad6eQ3PIARtveyPfSa787Sgc= 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:zysad6eQ3PIARtveyPfSa787Sgc='} 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078dd4ad8ad4671f2-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/ 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/ 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/ 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:/xryqqk-64/ 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g= 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:+GS1HoppR3V1MF0NIMM7imwyQ1g=', 'Content-Length': '0'} 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002efd4809b11462bd-006615d08d-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-64/obj185 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-64/obj185 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-64/obj185 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.808 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:34:37 GMT 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:/xryqqk-64/obj185 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:d24UjhFCugwq8imVSlFCHrmoHvI= 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:34:37 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:d24UjhFCugwq8imVSlFCHrmoHvI='} 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2f64299bf47d9de-006615d08d-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:34:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-64 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-64 2024-04-10T00:01:41.809 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-64 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:38.094+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:43.094+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:48.095+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:53.095+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:34:58.096+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:03.096+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:08.097+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:13.097+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:18.098+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:23.099+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:28.100+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:33.100+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:38.100+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:43.101+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:48.101+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:53.102+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:35:58.103+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:03.103+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.811 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:08.104+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:13.104+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:18.105+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:23.105+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:28.106+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:33.107+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.812 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:38.107+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:43.108+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:48.109+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:53.109+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:36:58.110+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:03.111+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:08.112+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:13.111+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.814 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:18.112+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:23.112+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:28.113+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:33.113+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:38.114+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:43.115+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:48.115+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:53.116+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:37:58.116+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:03.117+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:08.118+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:13.118+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:18.119+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:23.120+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:28.120+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:33.121+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:38.122+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:43.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:48.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:53.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:38:58.123+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:03.124+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:08.124+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:13.125+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:18.126+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:23.126+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:28.127+0000 7ffa4473b780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:33.127+0000 7ffa4473b780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:33.127+0000 7ffa4473b780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:33.127+0000 7ffa4473b780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.817 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.818 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.818 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.818 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-10T00:01:41.818 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1915, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-10T00:01:41.819 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/ 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/ 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/ 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.820 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:/xryqqk-65/ 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lXjh9TDaut8lf9TEmUlvLCt6U4c= 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lXjh9TDaut8lf9TEmUlvLCt6U4c=', 'Content-Length': '0'} 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005364a6309c2437a7-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=xryqqk-65 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.821 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.822 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705973.190683_1283.1", 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:39:33.190683Z" 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.823 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:33.433+0000 7f7ccfcab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:33.434+0000 7f7ccfcab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.824 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712705973.190683_1283.1", 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:39:33Z" 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.829 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.830 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "marker": "1_1712705973.190683_1283.1", 2024-04-10T00:01:41.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:33.190683Z", 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.833 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:39:33Z" 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712705973.190683_1283.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:33.190683Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712705973.190683_1283.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/ 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/ 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/ 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:/xryqqk-65/ 2024-04-10T00:01:41.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097ee899dfb62090e-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/obj11 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/obj11 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/obj11 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:/xryqqk-65/obj11 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Vac56wNqEZrHpnt6dzTuVrub77E= 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Vac56wNqEZrHpnt6dzTuVrub77E='} 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002ddb6fac8b9587f-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/ 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/ 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/ 2024-04-10T00:01:41.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:/xryqqk-65/ 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-10T00:01:41.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb1ed492ddb3789e-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/obj22 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/obj22 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/obj22 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.839 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:/xryqqk-65/obj22 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:uFTizUuWJG/M4DPUiSRQTyQtR6s= 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:uFTizUuWJG/M4DPUiSRQTyQtR6s='} 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce42c06247c16808-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/ 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/ 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/ 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:/xryqqk-65/ 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.841 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a2e3098d439cabc-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/obj33 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/obj33 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/obj33 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:/xryqqk-65/obj33 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MWt9sdnJgb5pDKsd28PPqHYx6PY= 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MWt9sdnJgb5pDKsd28PPqHYx6PY='} 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048514cbf01c73cfc-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/ 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/ 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/ 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:/xryqqk-65/ 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-10T00:01:41.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1e7cacf09cb5082-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/obj44 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/obj44 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/obj44 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:/xryqqk-65/obj44 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DZLXM9544MyZdELgD5ow+dZDXoE= 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DZLXM9544MyZdELgD5ow+dZDXoE='} 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1b2af5f4d685a29-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/ 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/ 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/ 2024-04-10T00:01:41.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:/xryqqk-65/ 2024-04-10T00:01:41.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA= 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:lqkd/NWYE44Exf1i/XPiC9lM/wA=', 'Content-Length': '0'} 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031c71aa1462f0d8a-006615d1b5-1b6d-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-65/obj55 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-65/obj55 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-65/obj55 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:39:33 GMT 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:/xryqqk-65/obj55 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:1c1LE0zRKncrA2Zvg1nrNmdpfZ4= 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:39:33 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:1c1LE0zRKncrA2Zvg1nrNmdpfZ4='} 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a587b77fcc240055-006615d1b5-1b6d-a1'), ('Date', 'Tue, 09 Apr 2024 23:39:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=xryqqk-65 2024-04-10T00:01:41.849 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: Sync is disabled for bucket xryqqk-65 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=xryqqk-65 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'xryqqk-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-10T00:01:41.850 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket xryqqk-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:33.830+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:38.831+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:43.831+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:48.832+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:53.832+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:39:58.833+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:03.834+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.851 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:08.834+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:13.835+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:18.836+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:23.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:28.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:33.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:38.837+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:43.838+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:48.839+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:53.839+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:40:58.839+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:03.840+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:08.841+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.852 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:13.841+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:18.842+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:23.842+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:28.843+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:33.843+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:38.844+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:43.844+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:48.845+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:53.846+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:41:58.846+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:03.848+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:08.848+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:13.848+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:18.849+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.853 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:23.849+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:28.850+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:33.850+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:38.851+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:43.852+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:48.852+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:53.853+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:42:58.853+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:03.854+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:08.854+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:13.855+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:18.855+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:23.856+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:28.857+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:33.858+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:38.858+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:43.860+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:48.859+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:53.859+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:43:58.860+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:03.861+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:08.861+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:13.862+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:18.862+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:23.863+0000 7f92802c1780 1 waiting to reach incremental sync.. 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:28.863+0000 7f92802c1780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:28.863+0000 7f92802c1780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:44:28.863+0000 7f92802c1780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-04-10T00:01:41.856 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:41.857 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:41.857 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:41.857 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2213, in test_sync_flow_symmetrical_zonegroup_all 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[1], content) 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:46:00.051842Z" 2024-04-10T00:01:41.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.860 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:16.843+0000 7f7cf7069780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:16.844+0000 7f7cf7069780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.861 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.862 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.863 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:16Z" 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.866 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.867 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.868 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:16Z" 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.870 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[]} 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.871 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[],"status":"allowed"}]} 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.872 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.873 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "epoch": 12, 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": [], 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_map": { 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zonegroups": [ 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a", 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a", 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": true, 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.874 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a1", 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:41.875 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:41.876 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a2", 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": false, 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:41.877 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.878 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [ 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-group", 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "data_flow": { 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "symmetrical": [ 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-flow1", 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.879 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pipes": [ 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-pipe", 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "*", 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:41.880 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": { 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "*", 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "params": { 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "filter": { 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [] 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.881 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": {}, 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "priority": 0, 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "mode": "system", 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user": "" 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "enabled" 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:41.882 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "b", 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b", 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": false, 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:41.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b1", 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:41.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b2", 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:41.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [] 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "short_zone_ids": [ 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2914326150 2024-04-10T00:01:41.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 89109471 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 3383989928 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2084093761 2024-04-10T00:01:41.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zonegroup": "a", 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_config": { 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_quota": { 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_quota": { 2024-04-10T00:01:41.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_ratelimit": { 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_ratelimit": { 2024-04-10T00:01:41.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "anonymous_ratelimit": { 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_name": "test-realm", 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-67 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-67/ 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-67/ 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-67/ 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:46:18 GMT 2024-04-10T00:01:41.894 INFO:tasks.rgw_multisite_tests:/xryqqk-67/ 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Mp7LSQX3BdyIXXnd5f6NMMW2Qew= 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Mp7LSQX3BdyIXXnd5f6NMMW2Qew=', 'Content-Length': '0'} 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000cf651e4fb1c0d6b3-006615d34a-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-67/ 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-67/ 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-67/ 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:46:18 GMT 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:/xryqqk-67/ 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5LV9wr2y/I8NI5WmVjC0L0Dxr6Q= 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5LV9wr2y/I8NI5WmVjC0L0Dxr6Q=', 'Content-Length': '0'} 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a7bdb1688ec4201-006615d34a-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-67/obj1 2024-04-10T00:01:41.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-67/obj1 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-67/obj1 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.897 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:46:18 GMT 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:/xryqqk-67/obj1 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:bjQcZaoP0t4MxEs25qXdmiay420= 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:bjQcZaoP0t4MxEs25qXdmiay420='} 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000205e7f7b368d6f0d-006615d34a-237a-a1'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=xryqqk-68 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-68/ 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-68/ 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-68/ 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:46:18 GMT 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:/xryqqk-68/ 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.899 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Ajb480v8fV8PW0zYIYdLldWQtr8= 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Ajb480v8fV8PW0zYIYdLldWQtr8=', 'Content-Length': '0'} 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000057943f7a2d5934e-006615d34a-237c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-68/ 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-68/ 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-68/ 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:46:18 GMT 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:/xryqqk-68/ 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:5z8F3SDMZv8phEaEFnpLRB0SMQg= 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:5z8F3SDMZv8phEaEFnpLRB0SMQg=', 'Content-Length': '0'} 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7ba13c4d3eb562f-006615d34a-237c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-68/obj2 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-68/obj2 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-68/obj2 2024-04-10T00:01:41.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:46:18 GMT 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:/xryqqk-68/obj2 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:41.902 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xI6TCrp+gCqCoivrbT7cPbPesek= 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:46:18 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xI6TCrp+gCqCoivrbT7cPbPesek='} 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001009301bd6468e31-006615d34a-237c-a2'), ('Date', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:41.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706378.392455_1368.1", 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:46:18.392455Z" 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:41.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:18.789+0000 7f840c3a1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:18.791+0000 7f840c3a1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:18Z" 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.915 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.916 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.917 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.918 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.919 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.920 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:18Z" 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:23.935+0000 7fc04804a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:23.936+0000 7fc04804a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:23Z" 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.926 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.927 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.928 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.929 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.930 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:23Z" 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:29.073+0000 7f1578cfa780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:29.074+0000 7f1578cfa780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.933 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.934 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.935 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.936 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:29Z" 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.937 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.938 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.939 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.940 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.941 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:29Z" 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.942 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:34.216+0000 7f9703467780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:34.217+0000 7f9703467780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.943 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:34Z" 2024-04-10T00:01:41.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.948 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.949 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.950 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.951 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.952 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:34Z" 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:39.352+0000 7f4edde9e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:39.353+0000 7f4edde9e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:39Z" 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.958 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.959 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.960 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.961 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.962 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:39Z" 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:44.490+0000 7fb191248780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:44.491+0000 7fb191248780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:44Z" 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.969 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.970 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.971 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:44Z" 2024-04-10T00:01:41.973 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:49.624+0000 7fbf456ae780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:49.625+0000 7fbf456ae780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:49Z" 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.979 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.980 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.983 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:49Z" 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:54.757+0000 7f3636761780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:54.758+0000 7f3636761780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:54Z" 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:41.990 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.991 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.992 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.993 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:41.994 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:54Z" 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.995 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:59.890+0000 7f8fdbcda780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:46:59.891+0000 7f8fdbcda780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.996 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.997 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.998 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:41.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:46:59Z" 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.001 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.002 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.003 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.004 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.005 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:46:59Z" 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.006 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:05.012+0000 7f2b4cb3a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:05.013+0000 7f2b4cb3a780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.007 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.008 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.009 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.010 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.011 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:05Z" 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.012 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.013 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.014 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.015 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.015 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.015 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.015 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.019 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.020 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.021 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:05Z" 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.022 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:10.135+0000 7f8abd3c1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:10.136+0000 7f8abd3c1780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.023 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.024 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.025 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.026 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:10Z" 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.028 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.029 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.030 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.031 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.032 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:10Z" 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:15.294+0000 7f502002f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:15.295+0000 7f502002f780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.035 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.036 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.037 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:15Z" 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.038 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.040 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.041 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.042 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.043 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:15Z" 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:20.420+0000 7f09e997d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:20.421+0000 7f09e997d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:20Z" 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.049 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.051 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.052 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.053 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.054 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:20Z" 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:25.563+0000 7f1a2b88b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:25.564+0000 7f1a2b88b780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:25Z" 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.061 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.062 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.063 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.065 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:25Z" 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.066 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:30.689+0000 7f728d075780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:30.690+0000 7f728d075780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.067 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.068 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:30Z" 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.072 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.073 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.074 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.075 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:30Z" 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.076 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:35.823+0000 7fa70c4e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:35.824+0000 7fa70c4e6780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.077 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.078 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.079 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.080 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.081 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.082 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.083 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:35Z" 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.084 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.085 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.086 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.087 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.088 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.089 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:35Z" 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:40.951+0000 7fafebfae780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:40.952+0000 7fafebfae780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:40Z" 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.096 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.097 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.098 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.099 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.100 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:40Z" 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:46.077+0000 7f8f831fd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:46.078+0000 7f8f831fd780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.106 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:46Z" 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.107 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.108 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.109 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.110 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:46Z" 2024-04-10T00:01:42.111 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi063.stdout: INFO: 2024-04-09T23:47:49.350+0000 7fc24521f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:51.213+0000 7fe379bab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.112 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:51.214+0000 7fe379bab780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.113 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.114 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.115 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.116 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.117 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.118 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.119 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.120 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.121 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:51Z" 2024-04-10T00:01:42.122 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.123 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.123 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.123 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.123 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.123 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.123 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.124 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.125 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.126 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.126 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.126 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.126 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.126 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.126 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.127 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.128 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:51Z" 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.129 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:56.970+0000 7fee4635e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:47:56.971+0000 7fee4635e780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.130 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.131 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.132 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.133 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.134 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.135 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:47:56Z" 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.136 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.137 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.138 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706360.051842_1327.1", 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:00.051842Z", 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.140 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:47:56Z" 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706360.051842_1327.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:00.051842Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706360.051842_1327.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706378.392455_1368.1 target=1_1712706360.051842_1327.1 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.141 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:02.109+0000 7f1e20a2c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:02.110+0000 7f1e20a2c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706378.392455_1368.1", 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.392455Z", 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:02Z" 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.147 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.148 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.149 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706378.392455_1368.1", 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.392455Z", 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.150 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.151 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:02Z" 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706378.392455_1368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.392455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.152 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.153 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.154 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.155 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.156 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:02Z" 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.159 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.160 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.161 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.162 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:02Z" 2024-04-10T00:01:42.163 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-67/ 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-67/ 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-67/ 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:/xryqqk-67/ 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wpzBXAA/E1N9VMbaHWV0KHYrpzw= 2024-04-10T00:01:42.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wpzBXAA/E1N9VMbaHWV0KHYrpzw=', 'Content-Length': '0'} 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fdb938b71fb67ed-006615d3b2-237c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-67/obj1 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-67/obj1 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-67/obj1 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:/xryqqk-67/obj1 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SOizCtI6mO5Cj7q+tYaYy+HU3FQ= 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SOizCtI6mO5Cj7q+tYaYy+HU3FQ=', 'Content-Length': '0'} 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-67:a5723776-f142-428c-a6df-d7bf949257da.9086.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000178fb8a8db0a35ab-006615d3b2-237c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-67/obj1 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-67/obj1 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-67/obj1 2024-04-10T00:01:42.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:/xryqqk-67/obj1 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.168 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IKXVa/ihegDgLRkYfQhHmw+Swtk= 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IKXVa/ihegDgLRkYfQhHmw+Swtk=', 'Content-Length': '0'} 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:46:18 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-67:a5723776-f142-428c-a6df-d7bf949257da.9086.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000807a3de531046285-006615d3b2-237c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-68/ 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-68/ 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-68/ 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-10T00:01:42.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:/xryqqk-68/ 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ImcKSFb6/Bcq/KZCmQojtxdjWTM= 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ImcKSFb6/Bcq/KZCmQojtxdjWTM=', 'Content-Length': '0'} 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad550daca097f858-006615d3b2-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-68/obj2 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-68/obj2 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-68/obj2 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:/xryqqk-68/obj2 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:BE92Ab5VMSrpOZ3ytQgJpCYHlTQ= 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:BE92Ab5VMSrpOZ3ytQgJpCYHlTQ=', 'Content-Length': '0'} 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d04349a6a226e3bd-006615d3b2-237a-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.172 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:42.173 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.173 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:42.173 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_single_bucket: 2024-04-10T00:01:42.173 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2465, in test_sync_single_bucket 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnames, content) 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-04-10T00:01:42.174 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706378.392455_1368.1", 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:46:18.392455Z" 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:02.593+0000 7f36bf180780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:02.594+0000 7f36bf180780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706378.392455_1368.1", 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.392455Z", 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:02Z" 2024-04-10T00:01:42.182 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.183 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.185 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706378.392455_1368.1", 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.392455Z", 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.186 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.187 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:02Z" 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706378.392455_1368.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.392455Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706378.392455_1368.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.188 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-69 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/ 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/ 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/ 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.190 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:/xryqqk-69/ 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:/l7EIfrYi4gEa24JvsFfp8ueS6c= 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:/l7EIfrYi4gEa24JvsFfp8ueS6c=', 'Content-Length': '0'} 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000ddadec9e145ef531-006615d3b2-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-70 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-70/ 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-70/ 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-70/ 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:02 GMT 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:/xryqqk-70/ 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ce05M61rBsndAHHRmvdXflDENZ0= 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:02 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ce05M61rBsndAHHRmvdXflDENZ0=', 'Content-Length': '0'} 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000619a293a99aa86c4-006615d3b2-237a-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.192 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:42.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706482.898476_1388.1", 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:48:02.898476Z" 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.194 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:03.124+0000 7f0c827e2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:03.126+0000 7f0c827e2780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706482.898476_1388.1", 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.199 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:03Z" 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.200 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706482.898476_1388.1", 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.203 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:03Z" 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.204 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706482.898476_1388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:02.898476Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.205 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a1","a2"]},"dest":{"bucket":"*","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=xryqqk-69', '--dest-bucket=xryqqk-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=xryqqk-69', '--dest-bucket=xryqqk-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=xryqqk-69 --dest-bucket=xryqqk-69 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a1","a2"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"xryqqk-69","zones":["a1","a2"]},"dest":{"bucket":"xryqqk-69","zones":["a1","a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "epoch": 13, 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": [], 2024-04-10T00:01:42.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_map": { 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zonegroups": [ 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a", 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a", 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": true, 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a1", 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a2", 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": false, 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [ 2024-04-10T00:01:42.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-group", 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "data_flow": { 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "symmetrical": [ 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-flow1", 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.214 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pipes": [ 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-pipe", 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "xryqqk-69", 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.215 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": { 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "xryqqk-69", 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "params": { 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "filter": { 2024-04-10T00:01:42.216 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [] 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": {}, 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "priority": 0, 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "mode": "system", 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user": "" 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.217 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "enabled" 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.218 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "b", 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b", 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": false, 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b1", 2024-04-10T00:01:42.219 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.220 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b2", 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:42.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [] 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "short_zone_ids": [ 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2914326150 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 89109471 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 3383989928 2024-04-10T00:01:42.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2084093761 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zonegroup": "a", 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_config": { 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_quota": { 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:42.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_quota": { 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_ratelimit": { 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:42.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_ratelimit": { 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:42.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "anonymous_ratelimit": { 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_name": "test-realm", 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"xryqqk-69"},"dest":{"zone":"a1","bucket":"xryqqk-69"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"xryqqk-69"},"dest":{"zone":"a2","bucket":"xryqqk-69"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/ 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/ 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/ 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:/xryqqk-69/ 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo= 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo=', 'Content-Length': '0'} 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e91a8df5f85901c-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj1 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj1 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj1 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.231 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj1 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:09VwPBeTCM99Ds9Cq2u5kFp/AtQ= 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:09VwPBeTCM99Ds9Cq2u5kFp/AtQ='} 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a39c695455254ad-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/ 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/ 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/ 2024-04-10T00:01:42.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:/xryqqk-69/ 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo= 2024-04-10T00:01:42.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo=', 'Content-Length': '0'} 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa32332e82b769b6-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj2 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj2 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj2 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.234 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj2 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:rNCuhyNB0ER1wAieZsj4aIwCLzU= 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:rNCuhyNB0ER1wAieZsj4aIwCLzU='} 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002b3c923e7e58f136-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/ 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/ 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/ 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:/xryqqk-69/ 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.236 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo= 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:IJcSTDtTAaIYJTUHTyTN+G8xIXo=', 'Content-Length': '0'} 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001dd2ce4d7e81ab24-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj3 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj3 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj3 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj3 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.238 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:e7J09uuvnzMRs4FIlnS4S7Ny7NQ= 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:e7J09uuvnzMRs4FIlnS4S7Ny7NQ='} 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000974e99a8755c0af3-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-70/ 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-70/ 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-70/ 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:/xryqqk-70/ 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:QrAoEjzjDWjGE5wlXa05nSKuyyM= 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:QrAoEjzjDWjGE5wlXa05nSKuyyM=', 'Content-Length': '0'} 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008854c339c8693280-006615d3b4-24a2-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-10T00:01:42.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:48:04 GMT 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:/xryqqk-70/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:AtR40feVszTUGUk4oXcu433avyY= 2024-04-10T00:01:42.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:48:04 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:AtR40feVszTUGUk4oXcu433avyY='} 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b22ba852d7c3e4f0-006615d3b4-24a2-a1'), ('Date', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706482.898476_1388.1", 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:48:02.898476Z" 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:04.633+0000 7f0f71e89780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:48:04.634+0000 7f0f71e89780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.245 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.246 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706482.898476_1388.1", 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.247 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.248 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.249 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:04Z" 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.250 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.251 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.252 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706482.898476_1388.1", 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.253 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:48:02.898476Z", 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.254 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:04Z" 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.255 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706482.898476_1388.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:02.898476Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706482.898476_1388.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023249", 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:48:04.194182Z" 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019723", 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:48:04.241192Z" 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018481", 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:48:04.136792Z" 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.257 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.260 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.261 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.262 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:04Z" 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.263 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.264 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.265 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.266 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.267 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:04Z" 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:10Z" 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.274 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.275 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.276 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.277 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.278 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:10Z" 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.279 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.280 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.281 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.282 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.283 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:15Z" 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.285 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.286 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.287 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.288 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:15Z" 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:20Z" 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.295 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.296 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.298 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:20Z" 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.300 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.301 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.302 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.303 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:25Z" 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.305 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.306 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.307 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.308 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:25Z" 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.310 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.314 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:30Z" 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.315 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.317 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.318 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:30Z" 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.321 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:35Z" 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.326 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.327 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.328 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.329 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:35Z" 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.330 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.332 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:40Z" 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.336 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.337 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.339 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:40Z" 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:46Z" 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.346 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.347 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.348 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.349 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:46Z" 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.350 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.351 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.352 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.353 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:51Z" 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.356 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.357 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.358 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.359 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:51Z" 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.360 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:48:56Z" 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.365 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.366 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.367 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.369 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.369 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.369 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.369 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.369 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.369 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.370 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:48:56Z" 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:01Z" 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.378 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.379 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.380 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.381 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.382 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.382 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.382 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.382 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.382 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.383 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:01Z" 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:06Z" 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.390 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.391 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.393 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.393 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.393 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.394 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.394 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.394 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:06Z" 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.396 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.397 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.397 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:11Z" 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.402 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.404 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.404 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.404 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.404 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.404 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.405 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:11Z" 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.407 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:16Z" 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.413 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.414 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.415 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.416 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:16Z" 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.417 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:22Z" 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.425 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.426 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.429 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:22Z" 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:27Z" 2024-04-10T00:01:42.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.436 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.437 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.438 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.439 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.440 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:27Z" 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.441 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.445 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.446 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:32Z" 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.447 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.448 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.449 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.450 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:32Z" 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.451 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.456 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:37Z" 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.457 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.458 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.459 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.460 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:37Z" 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.461 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.466 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:42Z" 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.467 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.468 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.469 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.470 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.471 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.472 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.472 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.472 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.472 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:42Z" 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.474 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.474 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.477 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.478 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.479 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:47Z" 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.480 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.481 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.482 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.483 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.484 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:47Z" 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.485 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.487 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.488 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.489 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.490 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:52Z" 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.495 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.496 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.496 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.496 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.496 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.496 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.496 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.497 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.497 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.497 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.497 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.498 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.498 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.498 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.498 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.499 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:52Z" 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:49:58Z" 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.507 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.508 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.510 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:49:58Z" 2024-04-10T00:01:42.511 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:50:03Z" 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.517 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.518 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000022969", 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:46:18.339197Z" 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.519 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019582", 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.108343Z" 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.520 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018339", 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:39:44.109973Z" 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.521 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:50:03Z" 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022969', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:46:18.339197Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019582', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.108343Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018339', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:39:44.109973Z'}}] 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000022969', 2: '00000000000000000000:00000000000000019582', 3: '00000000000000000000:00000000000000018339'} 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000023249 target=00000000000000000000:00000000000000022969, shard=2 master=00000000000000000000:00000000000000019723 target=00000000000000000000:00000000000000019582, shard=3 master=00000000000000000000:00000000000000018481 target=00000000000000000000:00000000000000018339 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.522 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023249", 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019723", 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018481", 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:50:08Z" 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.527 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011822", 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:34:58.104853Z" 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023249", 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:48:04.194182Z" 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019723", 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.529 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:48:04.241192Z" 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018481", 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:48:04.136792Z" 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:50:08Z" 2024-04-10T00:01:42.531 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011822', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:34:58.104853Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023249', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.194182Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019723', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.241192Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018481', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:48:04.136792Z'}}] 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011822', 1: '00000000000000000000:00000000000000023249', 2: '00000000000000000000:00000000000000019723', 3: '00000000000000000000:00000000000000018481'} 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/ 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/ 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/ 2024-04-10T00:01:42.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:42.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.534 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.534 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.535 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.535 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:50:08 GMT 2024-04-10T00:01:42.535 INFO:tasks.rgw_multisite_tests:/xryqqk-69/ 2024-04-10T00:01:42.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.535 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DrajcCcw39Gvn2PxDYGFFRPwiVc= 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DrajcCcw39Gvn2PxDYGFFRPwiVc=', 'Content-Length': '0'} 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f5f1be63cc77325-006615d430-24ae-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj1 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj1 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj1 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:50:08 GMT 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj1 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:ArZzu2xy0zwuzcZpi15tiVYYL2E= 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:ArZzu2xy0zwuzcZpi15tiVYYL2E=', 'Content-Length': '0'} 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af91cfa3dd69b2af-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj1 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj1 2024-04-10T00:01:42.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj1 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.538 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:50:08 GMT 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj1 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:55ZH7HTcc6CEIVavEf7ELvrYG8w= 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:55ZH7HTcc6CEIVavEf7ELvrYG8w=', 'Content-Length': '0'} 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000060cf0f9ff0dace25-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj2 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj2 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj2 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:50:08 GMT 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj2 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:tlhL9IfhR+wfxCTdhsisAvnORJ4= 2024-04-10T00:01:42.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:tlhL9IfhR+wfxCTdhsisAvnORJ4=', 'Content-Length': '0'} 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a67d6e664fcc0ec-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj2 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj2 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj2 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.541 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:50:08 GMT 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj2 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:cefVMWRO0rWiNUzZIDQZ6V9LmP8= 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:cefVMWRO0rWiNUzZIDQZ6V9LmP8=', 'Content-Length': '0'} 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:48:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-69:a5723776-f142-428c-a6df-d7bf949257da.9086.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ec1751902d8577b-006615d430-24ae-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-69/obj3 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-69/obj3 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-69/obj3 2024-04-10T00:01:42.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:50:08 GMT 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:/xryqqk-69/obj3 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.543 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:SNPiB75ri7QjEO/slJ8a5ylICc0= 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:50:08 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:SNPiB75ri7QjEO/slJ8a5ylICc0=', 'Content-Length': '0'} 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008c47bec9c3609e2d-006615d430-24ae-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:50:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_multiple_buckets_to_single: 2024-04-10T00:01:42.544 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:42.545 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-10T00:01:42.545 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-10T00:01:42.545 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-10T00:01:42.545 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2730, in test_sync_multiple_buckets_to_single 2024-04-10T00:01:42.545 INFO:tasks.rgw_multisite_tests: check_objects_exist(bucket, objnamesB, content) 2024-04-10T00:01:42.545 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2080, in check_objects_exist 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objname, content) 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:42.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706923.147638_1470.1", 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:23.147638Z" 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:23.481+0000 7f1cf61c5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:23.482+0000 7f1cf61c5780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:23Z" 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.553 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.554 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.555 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.556 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.557 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:23Z" 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:28.628+0000 7f3eca43c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:28.629+0000 7f3eca43c780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.559 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.560 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.561 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.562 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:28Z" 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.564 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.565 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.566 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.567 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.568 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.569 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:28Z" 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.570 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:33.768+0000 7f5aef209780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:33.769+0000 7f5aef209780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.571 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:33Z" 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.576 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.577 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.578 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.579 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.580 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:33Z" 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.581 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:38.890+0000 7f6333081780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:38.891+0000 7f6333081780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.582 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:38Z" 2024-04-10T00:01:42.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.587 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.588 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706906.897379_1465.1", 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.589 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:06.897379Z", 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.590 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:38Z" 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706906.897379_1465.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:06.897379Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.591 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706906.897379_1465.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706923.147638_1470.1 target=1_1712706906.897379_1465.1 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:44.018+0000 7ff9a8e3d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:44.020+0000 7ff9a8e3d780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.593 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.594 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706923.147638_1470.1", 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:23.147638Z", 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:44Z" 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.597 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.598 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706923.147638_1470.1", 2024-04-10T00:01:42.599 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:23.147638Z", 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.600 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:44Z" 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706923.147638_1470.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:23.147638Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706923.147638_1470.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[]} 2024-04-10T00:01:42.602 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-73 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/ 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/ 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/ 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:44 GMT 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:/xryqqk-73/ 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:0mhVWn7hdvUDXXhhL9otRZbgOzg= 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:0mhVWn7hdvUDXXhhL9otRZbgOzg=', 'Content-Length': '0'} 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000d429d4233ba8e6e0-006615d580-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.604 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=xryqqk-74 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/ 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/ 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/ 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.605 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:44 GMT 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:/xryqqk-74/ 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:xjrJH59bibVvrOe0aI4iP3EEM3U= 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:44 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:xjrJH59bibVvrOe0aI4iP3EEM3U=', 'Content-Length': '0'} 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000005a658ca698558098-006615d580-2703-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:42.606 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706944.197906_1478.1", 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:44.197906Z" 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.607 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:44.416+0000 7f4be70d9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:44.416+0000 7f4be70d9780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.608 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.609 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.610 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.611 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706944.165143_1471.1", 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:44.165143Z", 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.612 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.613 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:44Z" 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.614 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.615 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.616 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706944.165143_1471.1", 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:44.165143Z", 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.617 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.618 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:44Z" 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.165143_1471.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.165143Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.165143_1471.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=2 master=1_1712706944.197906_1478.1 target=1_1712706944.165143_1471.1 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.619 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:49.550+0000 7f3a5a524780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:49.551+0000 7f3a5a524780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.620 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.621 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.622 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706944.197906_1478.1", 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.623 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.624 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:49Z" 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.625 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.626 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.627 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.627 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.631 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.632 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706944.197906_1478.1", 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.633 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.634 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:49Z" 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-10T00:01:42.635 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-73', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-73', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xryqqk-73 --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-73 --dest-bucket=xryqqk-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-74', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.636 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-xryqqk-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=xryqqk-74', '--dest-bucket=xryqqk-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-xryqqk-74 --source-zones=a1 --dest-zones=a2 --source-bucket=xryqqk-74 --dest-bucket=xryqqk-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.637 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "epoch": 16, 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "predecessor_uuid": "a920d77d-da0e-43d1-9e1e-7c5d1538dcd0", 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": [], 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_map": { 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zonegroups": [ 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a", 2024-04-10T00:01:42.638 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a", 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": true, 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a1", 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.639 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.640 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "a2", 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi063.front.sepia.ceph.com:8001" 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": false, 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.641 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:42.642 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [ 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-group", 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "data_flow": { 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "directional": [ 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.643 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest_zone": "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pipes": [ 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-pipe-xryqqk-73", 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "xryqqk-73", 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": { 2024-04-10T00:01:42.644 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "xryqqk-74", 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "params": { 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "filter": { 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [] 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": {}, 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "priority": 0, 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "mode": "system", 2024-04-10T00:01:42.645 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user": "" 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "sync-pipe-xryqqk-74", 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "xryqqk-74", 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "a5723776-f142-428c-a6df-d7bf949257da" 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": { 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket": "xryqqk-74", 2024-04-10T00:01:42.646 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "e208d06e-fe81-4216-93c9-2cb12ee81a65" 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "params": { 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "source": { 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "filter": { 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [] 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "dest": {}, 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "priority": 0, 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "mode": "system", 2024-04-10T00:01:42.647 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user": "" 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "enabled" 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "b", 2024-04-10T00:01:42.648 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b", 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "api_name": "", 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "is_master": false, 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames": [], 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "hostnames_s3website": [], 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "zones": [ 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b1", 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.649 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8000" 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.650 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "id": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "b2", 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "endpoints": [ 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "http://smithi137.front.sepia.ceph.com:8001" 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_meta": false, 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "log_data": true, 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "read_only": false, 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tier_type": "", 2024-04-10T00:01:42.651 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from_all": true, 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_from": [], 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "redirect_zone": "", 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "supported_features": [ 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "compress-encrypted", 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "placement_targets": [ 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "name": "default-placement", 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "tags": [], 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "storage_classes": [ 2024-04-10T00:01:42.652 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "STANDARD" 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "default_placement": "default-placement", 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_policy": { 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "groups": [] 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled_features": [ 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "resharding" 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.653 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ], 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "short_zone_ids": [ 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "561bc780-8a4a-44a9-b0b2-d59ca12b2689", 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2914326150 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 89109471 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "a98f2146-689b-4f6e-84ee-fe9dc6a42ed3", 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 3383989928 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.654 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": "e208d06e-fe81-4216-93c9-2cb12ee81a65", 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": 2084093761 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zonegroup": "a", 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "master_zone": "a5723776-f142-428c-a6df-d7bf949257da", 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period_config": { 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_quota": { 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:42.655 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_quota": { 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "check_on_raw": false, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size": -1, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_size_kb": 0, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_objects": -1 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "user_ratelimit": { 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:42.656 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "bucket_ratelimit": { 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "anonymous_ratelimit": { 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_ops": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_ops": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_read_bytes": 0, 2024-04-10T00:01:42.657 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "max_write_bytes": 0, 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "enabled": false 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_id": "14ac2a00-44c3-4e52-ac2f-2b97df0062c3", 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_name": "test-realm", 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.658 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-xryqqk-73","source":{"bucket":"xryqqk-73","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-xryqqk-74","source":{"bucket":"xryqqk-74","zones":["a1"]},"dest":{"bucket":"xryqqk-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/ 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/ 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/ 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.659 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:/xryqqk-73/ 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY= 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY=', 'Content-Length': '0'} 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d69de0c2d3d021f-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/obj1 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/obj1 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/obj1 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:/xryqqk-73/obj1 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:z979YgZNneQ71Q5+Pv4xAOBz1SI= 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:z979YgZNneQ71Q5+Pv4xAOBz1SI='} 2024-04-10T00:01:42.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4c2faa846a418ef-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/ 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/ 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/ 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.662 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:/xryqqk-73/ 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY= 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:qAqHunjrBovnK9rovdPVQ8G2hdY=', 'Content-Length': '0'} 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005eaa4a83015bfa90-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/obj2 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/obj2 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/obj2 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.664 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.665 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.665 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.665 INFO:tasks.rgw_multisite_tests:/xryqqk-73/obj2 2024-04-10T00:01:42.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.665 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yrZ2KNOCLAXghNpCGenzsZ07Th4= 2024-04-10T00:01:42.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yrZ2KNOCLAXghNpCGenzsZ07Th4='} 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004b37303bfba3f917-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/ 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/ 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/ 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.666 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:/xryqqk-74/ 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8= 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8=', 'Content-Length': '0'} 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfb5c7328f7dd1a4-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj3 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj3 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj3 2024-04-10T00:01:42.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj3 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:y03dgm/u/HHy2rfnJ9Ix+MJcLho= 2024-04-10T00:01:42.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:y03dgm/u/HHy2rfnJ9Ix+MJcLho='} 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010dc7e1db445dff8-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/ 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/ 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/ 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.669 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:/xryqqk-74/ 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8= 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:MWQeMOv+h1qyTPeoKUbGCOpNwB8=', 'Content-Length': '0'} 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1839a45a7fff274-006615d586-2970-a1'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj4 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj4 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj4 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8000 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:PUT 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:55:50 GMT 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj4 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:m6nz43gWEu9sNC+MLyAS13hA/Zg= 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Tue, 09 Apr 2024 23:55:50 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:m6nz43gWEu9sNC+MLyAS13hA/Zg='} 2024-04-10T00:01:42.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e8992b5b1d79da06-006615d586-2970-a1'), ('Date', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: No --period given, using current period=feef76df-6795-4193-8fee-aa02d98d3595 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.055964Z" 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.672 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:19:26.074942Z" 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706944.197906_1478.1", 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:44.197906Z" 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T21:47:00.323078Z" 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.673 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:51.008+0000 7feb6adc8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: 2024-04-09T23:55:51.009+0000 7feb6adc8780 20 RGW-SYNC:meta: read sync status 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.674 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 1, 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.675 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.676 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712706944.197906_1478.1", 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.677 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "state": 1, 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "realm_epoch": 2 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.678 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 1, 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 1 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:51Z" 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests: "period": "feef76df-6795-4193-8fee-aa02d98d3595", 2024-04-10T00:01:42.679 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.055964_540.1", 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.055964Z", 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.680 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "marker": "1_1712704766.074942_528.1", 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:19:26.074942Z", 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.681 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "marker": "1_1712706944.197906_1478.1", 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:44.197906Z", 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.682 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "marker": "1_1712699220.323078_81.1", 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T21:47:00.323078Z", 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.683 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:51Z" 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712704766.055964_540.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.055964Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712704766.074942_528.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:19:26.074942Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712706944.197906_1478.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:44.197906Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712699220.323078_81.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T21:47:00.323078Z', 'realm_epoch': 2}}] 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712704766.055964_540.1', 1: '1_1712704766.074942_528.1', 2: '1_1712706944.197906_1478.1', 3: '1_1712699220.323078_81.1'} 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.684 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: [ 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023809", 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:50.552935Z" 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.685 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018901", 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "last_update": "2024-04-09T23:55:50.522415Z" 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.686 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.687 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.688 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.689 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.690 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:51Z" 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.691 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.692 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.693 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.694 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:51Z" 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.695 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.696 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.697 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.698 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.699 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.700 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:55:56Z" 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.701 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.702 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.703 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.704 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:55:56Z" 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.705 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.706 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.707 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.708 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.709 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:01Z" 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.710 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.711 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.712 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.713 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.714 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:01Z" 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.715 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.716 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.717 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.718 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.719 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:06Z" 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.720 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.721 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.722 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.723 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:06Z" 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.724 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.725 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.726 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.727 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.728 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.729 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:11Z" 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.730 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.731 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.732 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.733 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:11Z" 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.734 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.735 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.736 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.737 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.738 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:17Z" 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.739 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.740 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.741 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.742 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.743 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:17Z" 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.744 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.745 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.746 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.747 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:22Z" 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.749 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.750 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.751 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.752 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:22Z" 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.753 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:27Z" 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.759 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.760 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.762 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:27Z" 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.767 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.768 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:32Z" 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.769 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.770 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.771 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.772 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:32Z" 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.773 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.774 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.775 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.776 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.777 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.778 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:37Z" 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.779 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.780 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.781 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.782 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.783 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.784 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:37Z" 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.785 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.786 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.787 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.789 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:42Z" 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.791 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.792 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.793 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.794 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:42Z" 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.795 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.796 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.797 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.798 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.799 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:47Z" 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.800 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.801 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.802 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.803 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.804 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:47Z" 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.805 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.806 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.808 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.809 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:53Z" 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.810 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.811 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.812 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.813 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:53Z" 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.814 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.815 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.816 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.817 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.818 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.819 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:56:58Z" 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.820 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.821 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.822 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.823 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:56:58Z" 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.824 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.825 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.828 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:57:03Z" 2024-04-10T00:01:42.829 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.830 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.831 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.832 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.833 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:57:03Z" 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.834 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.835 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.836 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.837 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.838 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:57:08Z" 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.839 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.840 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.841 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.842 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.843 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:57:08Z" 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.844 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.847 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.848 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:57:13Z" 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.849 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.850 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.851 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.852 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:57:13Z" 2024-04-10T00:01:42.853 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.854 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.855 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.856 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.857 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.858 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:57:18Z" 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.859 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000011963", 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.137797Z" 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023529", 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.860 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.298968Z" 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.861 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018761", 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:54:24.309244Z" 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.862 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:57:18Z" 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011963', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.137797Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023529', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.298968Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018761', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:54:24.309244Z'}}] 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000011963', 1: '00000000000000000000:00000000000000023529', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018761'} 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000012103 target=00000000000000000000:00000000000000011963, shard=1 master=00000000000000000000:00000000000000023809 target=00000000000000000000:00000000000000023529, shard=3 master=00000000000000000000:00000000000000018901 target=00000000000000000000:00000000000000018761 2024-04-10T00:01:42.863 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "sync_status": { 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "info": { 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "sync", 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "num_shards": 4, 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "instance_id": 11300205680425452250 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "markers": [ 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 0, 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 9, 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 1, 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000023809", 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 57, 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 2, 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.866 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: { 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "key": 3, 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "val": { 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "status": "incremental-sync", 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "marker": "00000000000000000000:00000000000000018901", 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "next_step_marker": "", 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total_entries": 0, 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "pos": 0, 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-10T00:01:42.867 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: ] 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "full_sync": { 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "total": 66, 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "complete": 66 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: }, 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: "current_time": "2024-04-09T23:57:23Z" 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi063.stdout: INFO: } 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-10T00:01:42.868 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "instance_id": 11300205680425452250 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000012103", 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:50.606072Z" 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.869 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000023809", 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:50.552935Z" 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-10T00:01:42.870 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000019863", 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:21.139072Z" 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: { 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000018901", 2024-04-10T00:01:42.871 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-09T23:55:50.522415Z" 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: } 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: ] 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: }, 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-09T23:57:23Z" 2024-04-10T00:01:42.872 INFO:tasks.rgw_multisite_tests:} 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012103', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.606072Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023809', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.552935Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019863', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:21.139072Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018901', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-09T23:55:50.522415Z'}}] 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000012103', 1: '00000000000000000000:00000000000000023809', 2: '00000000000000000000:00000000000000019863', 3: '00000000000000000000:00000000000000018901'} 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/ 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/ 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/ 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:/xryqqk-73/ 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:APD2UEG4mEbmJjKRCnQebZI/puo= 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:APD2UEG4mEbmJjKRCnQebZI/puo=', 'Content-Length': '0'} 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049de10d8fcd13c17-006615d5e3-297c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/obj1 2024-04-10T00:01:42.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/obj1 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/obj1 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.875 INFO:tasks.rgw_multisite_tests:/xryqqk-73/obj1 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DfN/a70mmtdF5qmDet3SjjfHNBY= 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DfN/a70mmtdF5qmDet3SjjfHNBY=', 'Content-Length': '0'} 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c1f6c9035a35fb4f-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/obj2 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/obj2 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/obj2 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:/xryqqk-73/obj2 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:kvodPwfiui4WB4EKH1HfEhksZaU= 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:kvodPwfiui4WB4EKH1HfEhksZaU=', 'Content-Length': '0'} 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f602f698a10f200f-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/obj3 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/obj3 2024-04-10T00:01:42.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/obj3 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.878 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:/xryqqk-73/obj3 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:wADqwrP4nEljfMG4nqwtq0gfqfQ= 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:wADqwrP4nEljfMG4nqwtq0gfqfQ=', 'Content-Length': '0'} 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e3008a3664e55096-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-73/obj4 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-73/obj4 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-73/obj4 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:/xryqqk-73/obj4 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:3mSdDhWVtJjJVAeVfw00MAqzBYw= 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:3mSdDhWVtJjJVAeVfw00MAqzBYw=', 'Content-Length': '0'} 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000443530aaef48defd-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/ 2024-04-10T00:01:42.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/ 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/ 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.881 INFO:tasks.rgw_multisite_tests:/xryqqk-74/ 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:yl4idAmtktXujJFB+T2ZEl2xUbA= 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:yl4idAmtktXujJFB+T2ZEl2xUbA=', 'Content-Length': '0'} 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ee5f2c3f9b3d02e-006615d5e3-297c-a2'), ('Content-Length', '0'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj1 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj1 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj1 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj1 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:93eStWR0Gb/FoKMjK4zHXgcsl4c= 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:93eStWR0Gb/FoKMjK4zHXgcsl4c=', 'Content-Length': '0'} 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002fa0981e72444877-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj1 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj1 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:42.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj1 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj1 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.884 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:DJwM+OSEJPTqK4BAjy9kjpSIIdE= 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:DJwM+OSEJPTqK4BAjy9kjpSIIdE=', 'Content-Length': '0'} 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f666c163ceb8655-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj2 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj2 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj2 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.885 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj2 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:Lejb2gMr09li79weeCr/v6jQzHw= 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:Lejb2gMr09li79weeCr/v6jQzHw=', 'Content-Length': '0'} 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8eb980382e78cf9-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj2 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj2 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj2 2024-04-10T00:01:42.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:GET 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj2 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:vE9rIiYAdv9JECb3mtc4lyWnPBU= 2024-04-10T00:01:42.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:vE9rIiYAdv9JECb3mtc4lyWnPBU=', 'Content-Length': '0'} 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Tue, 09 Apr 2024 23:55:50 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a5723776-f142-428c-a6df-d7bf949257da:xryqqk-73:a5723776-f142-428c-a6df-d7bf949257da.9991.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000031712246af3eb1e3-006615d5e3-297c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/xryqqk-74/obj3 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/xryqqk-74/obj3 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /xryqqk-74/obj3 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi063.front.sepia.ceph.com:8001 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-10T00:01:42.888 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:Tue, 09 Apr 2024 23:57:23 GMT 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:/xryqqk-74/obj3 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:AWS N5OV6KWZ8WWR3JMWMG4W:OZCy6q1KIhQ4LdVnI547SVlvLaU= 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Tue, 09 Apr 2024 23:57:23 GMT', 'Authorization': 'AWS N5OV6KWZ8WWR3JMWMG4W:OZCy6q1KIhQ4LdVnI547SVlvLaU=', 'Content-Length': '0'} 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007503ac52b4400943-006615d5e3-297c-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Tue, 09 Apr 2024 23:57:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.889 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-10T00:01:42.890 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 8214.516s 2024-04-10T00:01:42.890 INFO:tasks.rgw_multisite_tests: 2024-04-10T00:01:42.890 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, errors=1, failures=22) 2024-04-10T00:01:42.897 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-10T00:01:43.127 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=29b2e41755354f97afa4d91c15d00836 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-10T00:01:43.128 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-04-10T00:01:43.138 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-04-10T00:01:43.172 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-10T00:01:43.230 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-10T00:01:43.230 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:01:43.234 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-10T00:01:43.231+0000 7ff9034c1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 48171) UID: 0 2024-04-10T00:01:43.234 INFO:tasks.rgw.c1.client.0.smithi063.stdout:2024-04-10T00:01:43.231+0000 7ff9086967c0 -1 shutting down 2024-04-10T00:01:43.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-10T00:01:49.332 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-10T00:01:49.332 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-04-10T00:01:49.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/vault-root-token 2024-04-10T00:01:49.429 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-10T00:01:49.429 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:01:49.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2024-04-10T00:01:49.471 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-10T00:01:49.472+0000 7fc257243700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 105722) UID: 0 2024-04-10T00:01:49.471 INFO:tasks.rgw.c1.client.1.smithi063.stdout:2024-04-10T00:01:49.472+0000 7fc25c4187c0 -1 shutting down 2024-04-10T00:01:54.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-10T00:01:54.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-10T00:01:55.531 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-10T00:01:55.531 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-04-10T00:01:55.553 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/vault-root-token 2024-04-10T00:01:55.626 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-10T00:01:55.627 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:01:55.628 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-04-10T00:01:55.629+0000 7fa810dc9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 48070) UID: 0 2024-04-10T00:01:55.628 INFO:tasks.rgw.c2.client.0.smithi137.stdout:2024-04-10T00:01:55.629+0000 7fa815f9e7c0 -1 shutting down 2024-04-10T00:02:00.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-10T00:02:00.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-10T00:02:01.729 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-10T00:02:01.730 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-04-10T00:02:01.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/vault-root-token 2024-04-10T00:02:01.900 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-10T00:02:01.900 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:01.941 INFO:tasks.rgw.c2.client.1.smithi137.stdout:2024-04-10T00:02:01.941+0000 7ff8b7b67700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 49338) UID: 0 2024-04-10T00:02:01.941 INFO:tasks.rgw.c2.client.1.smithi137.stdout:2024-04-10T00:02:01.941+0000 7ff8bcd3c7c0 -1 shutting down 2024-04-10T00:02:06.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-10T00:02:06.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-10T00:02:08.002 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-10T00:02:08.002 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-04-10T00:02:08.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/vault-root-token 2024-04-10T00:02:08.095 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-10T00:02:08.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-04-10T00:02:08.696 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-10T00:02:08.697 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-10T00:02:08.697 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-04-10T00:02:08.697 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:08.792 INFO:tasks.ceph.c2.osd.0:Stopped 2024-04-10T00:02:08.792 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-04-10T00:02:08.793 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:08.872 INFO:tasks.ceph.c2.osd.1:Stopped 2024-04-10T00:02:08.872 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-04-10T00:02:08.872 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:08.925 INFO:tasks.ceph.c2.osd.2:Stopped 2024-04-10T00:02:08.926 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-10T00:02:08.926 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-04-10T00:02:08.926 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:08.951 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-04-10T00:02:08.951 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-10T00:02:08.951 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-04-10T00:02:08.952 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:09.014 INFO:tasks.ceph.c2.mon.a:Stopped 2024-04-10T00:02:09.015 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-10T00:02:09.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-10T00:02:09.132 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi137.front.sepia.ceph.com 2024-04-10T00:02:09.132 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-04-10T00:02:09.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi137.front.sepia.ceph.com 2024-04-10T00:02:09.474 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-04-10T00:02:09.620 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi137.front.sepia.ceph.com 2024-04-10T00:02:09.621 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-04-10T00:02:09.768 INFO:tasks.ceph:Archiving mon data... 2024-04-10T00:02:09.770 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/data/c2.mon.a.tgz 2024-04-10T00:02:09.770 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-04-10T00:02:09.787 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.wUg5wEyAWv 2024-04-10T00:02:09.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.wUg5wEyAWv 2024-04-10T00:02:10.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.wUg5wEyAWv 2024-04-10T00:02:10.142 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.wUg5wEyAWv is 502KB 2024-04-10T00:02:10.243 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.wUg5wEyAWv 2024-04-10T00:02:10.259 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-10T00:02:10.260 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-10T00:02:10.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-10T00:02:10.326 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-10T00:02:10.371 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-04-10T00:02:10.883 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-10T00:02:10.883 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-10T00:02:10.884 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-04-10T00:02:10.884 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:10.953 INFO:tasks.ceph.c1.osd.0:Stopped 2024-04-10T00:02:10.953 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-04-10T00:02:10.953 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:11.004 INFO:tasks.ceph.c1.osd.1:Stopped 2024-04-10T00:02:11.005 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-04-10T00:02:11.005 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:11.067 INFO:tasks.ceph.c1.osd.2:Stopped 2024-04-10T00:02:11.067 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-10T00:02:11.067 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-04-10T00:02:11.068 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:11.093 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-04-10T00:02:11.093 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-10T00:02:11.093 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-04-10T00:02:11.093 INFO:teuthology.orchestra.run:waiting for 300 2024-04-10T00:02:11.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2024-04-10T00:02:11.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~0s 2024-04-10T00:02:11.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2024-04-10T00:02:11.157 INFO:tasks.ceph.c1.mon.a:Stopped 2024-04-10T00:02:11.157 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-10T00:02:11.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-10T00:02:11.268 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-10T00:02:11.268 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-04-10T00:02:11.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-10T00:02:11.669 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-10T00:02:11.669 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-04-10T00:02:11.770 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-10T00:02:11.770 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-04-10T00:02:11.864 INFO:tasks.ceph:Archiving mon data... 2024-04-10T00:02:11.865 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/data/c1.mon.a.tgz 2024-04-10T00:02:11.865 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-04-10T00:02:11.883 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.kiWRT7zB9G 2024-04-10T00:02:11.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.kiWRT7zB9G 2024-04-10T00:02:12.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.kiWRT7zB9G 2024-04-10T00:02:12.236 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.kiWRT7zB9G is 556KB 2024-04-10T00:02:12.334 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.kiWRT7zB9G 2024-04-10T00:02:12.351 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-10T00:02:12.351 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-10T00:02:12.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-10T00:02:12.426 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-10T00:02:12.426 DEBUG:teuthology.orchestra.run.smithi063:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-10T00:02:12.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-10T00:02:12.486 INFO:teuthology.orchestra.run.smithi063.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-10T00:02:12.498 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-10T00:02:12.505 INFO:tasks.ceph:Archiving crash dumps... 2024-04-10T00:02:12.507 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/remote/smithi063/crash 2024-04-10T00:02:12.508 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-10T00:02:12.560 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/remote/smithi137/crash 2024-04-10T00:02:12.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-10T00:02:12.592 INFO:tasks.ceph:Compressing logs... 2024-04-10T00:02:12.593 DEBUG:teuthology.orchestra.run.smithi063:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T00:02:12.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T00:02:16.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 has been restored 2024-04-10T00:02:16.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 has been restored 2024-04-10T00:02:16.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a has been restored 2024-04-10T00:30:39.772 INFO:tasks.ceph:Archiving logs... 2024-04-10T00:30:39.787 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/remote/smithi063/log 2024-04-10T00:30:39.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-10T00:31:21.206 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/remote/smithi137/log 2024-04-10T00:31:21.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-10T00:34:45.376 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-10T00:34:45.387 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-10T00:34:45.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-10T00:34:45.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-10T00:34:45.512 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-10T00:34:45.522 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-10T00:34:45.522 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-10T00:34:45.525 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-10T00:34:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:bash: ntpq: command not found 2024-04-10T00:34:45.544 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2024-04-10T00:34:45.565 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-10T00:34:45.566 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-04-10T00:34:45.566 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 -16us[ -16us] +/- 80ms 2024-04-10T00:34:45.566 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -712us[ -711us] +/- 61ms 2024-04-10T00:34:45.566 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1044us[+1044us] +/- 82ms 2024-04-10T00:34:45.567 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-10T00:34:45.597 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-10T00:34:45.598 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-04-10T00:34:45.598 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 -55us[ -55us] +/- 80ms 2024-04-10T00:34:45.598 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -749us[ -746us] +/- 61ms 2024-04-10T00:34:45.598 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 101 +999us[+1001us] +/- 81ms 2024-04-10T00:34:45.598 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-10T00:34:45.599 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T00:34:45.609 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T00:34:45.610 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T00:34:45.638 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T00:34:45.757 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T00:34:45.880 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-10T00:34:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T00:34:45.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-10T00:34:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T00:34:45.948 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T00:34:45.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T00:34:45.978 INFO:teuthology.task.internal:Duration was 11494.424002 seconds 2024-04-10T00:34:45.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T00:34:46.012 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T00:34:46.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T00:34:46.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T00:34:46.081 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T00:34:46.098 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T00:34:46.553 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T00:34:46.553 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-04-10T00:34:46.553 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T00:34:46.582 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-04-10T00:34:46.582 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T00:34:46.609 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T00:34:46.609 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T00:34:46.625 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T00:34:46.713 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T00:34:46.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T00:34:46.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T00:34:46.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T00:34:46.833 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T00:34:46.834 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T00:34:46.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T00:34:46.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T00:34:46.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T00:34:46.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T00:34:46.939 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-04-10T00:34:46.972 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-04-10T00:34:46.987 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T00:34:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T00:34:47.021 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T00:34:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T00:34:47.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T00:34:47.055 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T00:34:47.056 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/remote/smithi063 2024-04-10T00:34:47.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T00:34:47.100 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917/remote/smithi137 2024-04-10T00:34:47.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T00:34:47.141 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T00:34:47.142 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T00:34:47.144 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T00:34:47.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T00:34:47.203 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T00:34:47.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T00:34:47.229 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T00:34:47.229 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T00:34:47.232 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T00:34:47.246 INFO:teuthology.orchestra.run.smithi063.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 00:34 /home/ubuntu/cephtest 2024-04-10T00:34:47.250 INFO:teuthology.orchestra.run.smithi137.stdout: 262158 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 00:34 /home/ubuntu/cephtest 2024-04-10T00:34:47.250 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T00:34:47.264 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T00:34:47.329 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T00:34:47.357 DEBUG:teuthology.nuke:shortname: smithi063 2024-04-10T00:34:47.357 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T00:34:47.385 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-09 21:17:27.270827', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP6JmiQup3jBbUc4fDfxA1Cc87n92MyW1E37W5w047Bu63ZvbHk+9o3MyQjRiYMiRUTXFQavy0sy2i0bbdnktEc='} 2024-04-10T00:34:47.389 DEBUG:teuthology.nuke:shortname: smithi137 2024-04-10T00:34:47.389 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T00:34:47.408 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_01:21:36-rgw-reef-release-distro-default-smithi/7647917', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-09 21:17:27.272267', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE/q3UvERIx3e325Ay3goK8EiCuOC295xGHManCHbnvw67Gvt8s2Nmjbhmw1zIntD9LFijFSyuVpPD5AtVlfbmU='} 2024-04-10T00:34:47.438 INFO:teuthology.orchestra.console.smithi063:Power off 2024-04-10T00:34:47.439 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:34:47.467 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-04-10T00:34:47.468 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:34:47.505 INFO:teuthology.orchestra.console.smithi137:Power off 2024-04-10T00:34:47.505 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:34:47.519 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-04-10T00:34:47.531 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-04-10T00:34:47.531 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:34:47.561 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-04-10T00:34:51.520 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:34:51.563 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:34:51.636 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-04-10T00:34:51.682 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-04-10T00:34:55.639 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:34:55.683 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:34:55.753 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-04-10T00:34:55.800 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-04-10T00:34:55.855 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-04-10T00:34:55.902 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-04-10T00:34:56.007 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} duration: 11494.424001932144 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=29b2e41755354f97afa4d91c15d00836 status: fail success: false 2024-04-10T00:34:56.008 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T00:34:56.100 INFO:teuthology.run:FAIL